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A NETWORK 

PAr^ftPnUNH nPTHg INVENTION 

FiaiH nf thp Invention 

The invention relates to data sharing and. more particularly, to sharing of 

multimedia data over a network. 

Computer technology is increasingly incorporated by musicians and multimedia 
production specialists to aide in the creative process. For example, musicians use 
computers configured as "sequencers" or "DAWs" (digital audio workstations) to record 
multimedia source material. such.as digital audio, digital video, and Musical Instrument 
Digital interface (MIDI) data. Sequences and DAWs then create sequence data to 
enable the user to se.ect.and edit various portions of the recorded data to produce a 
finished product. 

Sequencer software is often used when multiple artists collaborate m a project 
usually in the form of multitrack recordings of individual instruments gathered together 
in a recording studio. A production specialist then uses the sequencer software to edit 
the various tracks, both individually and in groups, to produce the final arrangement for 
theproduct. Often in a recording session, multiple lakes" of the same portion of music 
will be.recorded. enabling the production specialist* select the best portions of various 
' takes. Additional takes can be made during the session if necessary. 

Such collaboration is, of course, most convenient when all artists are present in 
the same location a. the same time. However, this is often not possible. For example, 
an orchestra can be assembled at a recording studio in Los Angeles but the vocalist 
may be in New York or London and thus unable to participate in person in the session, 
„ is, of course, possible for the vocalist to participate from a remote studio linked to the 
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main studio in Los Angeles by wide bandwidth, high fidelity communications channels. 
However, this is often prohibitively expensive, if not impossible. 

Various methods of overcoming this problem are known in the prior art. For 
example, the Res Rocket system of Rocket Networks, Inc. provides the ability for 
geographically separated users to share MIDI data over the Internet. However, 
professional multimedia production specialists commonly use a small number of widely 
known professional sequencer software packages. Since they have extensive 
experience in using the interface of a particular software package, they are often 
unwilling to forego the benefits of such experience to adopt an unfamiliar sequencer. 

It is therefore desirable to provide a system and method for professional artists 
and multimedia production specialists to collaborate from geographically separated 
locations using familiar user interfaces of existing sequencer software. 

SUMMARY OF THE INVENTION 

Features and advantages of the invention will be set forth fn the description 
which follows, and in part will be apparent from the description, or may be learned by 
practice of the invention. The objectives and other advantages of the invention will be 
realized and attained by the systems and methods particularly pointed out in the written 
description and claims hereof, as well as the appended drawings. 

In accordance with the purpose of the invention as embodied and broadly 

described..thejny^ritjr^^gLe_s apparatus for sharing seq uence data between a lodal 
sequencer station and at least one remote sequencer station over a network via a 
server, the sequence data representing audiovisual occurrences each having 
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descriptive characteristics and time characteristics. The apparatus inCudes a firs, 
interface modute receiving commands from a iooa. sequencer station and a da* 
packing modute coupied to the first interface moduie. The data packing moduie 
spends .0 the received commands by encapsutating sequence data from the .oca, 
sequencer station into broadcast date un«s retaining the descriptive characterises and 
fi.e reiafionshipsof.he sequence data. The date packaging modute aisoexbacts 
sequence date from broadcast date unKs receded fromthe serverfo, access by the 

to the firs, interface moduie and me date packaging modute. The broadcast handier 
presses commands received via thefirs, interface mod^. The apparatus atso 
incudes a server communicafions moduie responding ,o commands processed by the 

teas , one remote sequencer stefion. tee server communicafions moduie aiso receiving 

tedher incudes a notion queue handter coupted tethe server communicafions 
nrod u te and responsive ,o receipt o, date avaiiabte messages and broadcas, date unite 
fram tee server ,o transmfi nofificafions to the firs, interface for access by the iocai 
sequencer terminal. 

,n another aspect me invention provides a method for sharing sequence date 
between a iooa, sequencer stefion and at teas, one remote sequencer stefion over a 
nefivork via a se^er. tee sequence date represenfing audiovisua, occurences each 
Kevingdescripfive chateCensficsand^characterfsfics. The method inCudes 
,ece*ing commands via a Cien, appiicafion component from a user a, a loca. 

3' 
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sequencer station; responding to the received commands by encapsulating sequence 
data from the local sequencer station into broadcast data units retaining the descriptive 
characteristics and time relationships of the sequence data and transmitting broadcast 
data units to the server for distribution to at least one remote sequencer station; 

5 receiving data available messages from the server responding to receipt of data 
available messages from the server to transmit notifications to the client application 
component; responding to commands received from the client application component to 
request download of broadcast data units from the server, and receiving broadcast data 
units from the server and extracting sequence data from the received broadcast data 

10 units for access by the client application component. ; 

It is to be understood that both the foregoing general description and the 
following detailed description are exemplarily and explanatory and are intended to 
provide further explanation of the invention as claimed. 

The accompanying drawings are included to provide a further understanding of 

1 5 the invention and are incorporated in and constitute a part of this specification to 
illustrate embodiments of the invention and, together with the description, serve to 
explain the principles of the invention. 
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nnH nc tmp DRAWINGS 
Th, accompan^drawings which are incorporated in and oonamute a part of 

M specifleabon iilustrate embe— - !• " «*• * *" 
description serve to e,p,ain the objects advances and prints o, .he invention. 

In the drawings: 

embodiment olthepreaent invention; 

Rg zisablockdiagramahowingmodulesof.heae.iceaoomponen.ofFrg.l; 

Ra . 3 tsa dfcgram showing fhe hierarohicai retationship o, broadcast data un«s 

of the system of Fig. 1; 

Fig. « is a diagram showing me — «P*— « *»^— 4 *^ ** 
Track objects of the system of Fig. 1; 

Fig. 5 re e diagram showing the hetween Tree* ob^ and Event 

objects of the system of Fig. 1 ; 

Fig. 6 is a diagram showing the relationship between Assst objects and 

Rendering objects of the system of Fig. 1: 

F*. 7 tea diagram showing the reiationsh.p between dp object end Asset 

objects of the system of Fig. 1; 

Fig. 8 is a diagram showing the reiatfcnship between Event objec*, C» Event 
cbiecU.Ciip obiecte, and Asset obiedsofthe system of Fig. 1i ^ . 

objects, and Timeline objects of the system of Fig. 1: 
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Fig. 10 is a diagram showing the relationship of Project objects and Custom 
objects of the system of Fig . 1 ; and 

Fig. 11 is a diagram showing the relationship between Rocket objects, and 
Custom and Extendable objects of the system of Fig. 1 . 

DETAILED DESCRIPTION OF THE PREFER RED EMBODIMENTS 

Computer applications for musicians and multimedia production specialists 
(typically sequencers and DAWs) are built to allow users to record and edit multimedia 
data to create a multimedia project. Such applications are inherently single-purpose, - 
single-user applications. The present invention enables geographically separated 
persons operating individual sequencers and DAWs to collaborate. 

The basic paradigm of the present invention is.that of a "virtual studio." This, like 
a real-world studio, is a "place" for people to "meef and work on multimedia projects 
together. However, the people that an individual user works with in this virtual studio 
can be anywhere in the world - connected by a computer network. 

Fig. 1 shows a system 10 consistent with the present invention. System 10 

includes a server 12, a local sequencer station 14, and a plurality of remote sequencer 

stations 16, all interconnected via a network 18. Network 18 may be the Internet or 

may be a proprietary network. 

Local and remote sequencer stations 14 and 16 are preferably personal 

-computere,-such-as-Apple-PowerMacintoshes or^e ntium-based personal com putere 
running a version of the Windows operating system. Local and remote sequencer 
stations 14 and 16 include a client application component 20 preferably comprising a 

6 
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sequencer software package, or "sequencer." As noted above, sequencers create 
sequence data representing multimedia data which in turn represents audiovisual 
occurrences each having descriptive characteristics and time characteristics. 
Sequencers further enable a user to manipulate and edit the sequence data to generate 
multimedia products. Examples of appropriate sequencers include Lp^icAudio from 
Emagic Inc. of Grass Valley, California; Cubase from. Steinberg Soft- und.Hardware 
GmbH of Hamburg, Germany; and PjoTooJs from Digidesign, Inc. of Palo Alto, CA. 

Local sequencer station 14 and remote sequencer stations 16 may be, but are 
not required to-be, identical, and typically include display hardware such as a CRT and 
sound card (not shown) to provide audio and video output.. 

Local sequencer station 14 also includes a connection control component 22 
which allows a user at local sequencer station 14 to "log in" to server 12, navigate to a 
virtual studio, find other collaborators at remote sequencer stations 16, and 
communicate with those collaborators. Each-client application. component. 20_at local 
and remote sequencer stations 14 and 16 is able to load a project stored in the virtual 
studio, much as if it were created by the client application component at that station - 
but with some important differences. 

Client application components 20 typically provide an "arrangement" window on 
a display screen containing a plurality of "tracks," each displaying a track name, record 
status, channel assignment, and other simitar information. Consistent with the present 
invention,-the arrangement window also displays a new item: user name. The user 
name is the name of the individual that "owns" that particular track, after creating it on 
his local sequencer station. This novel concept indicates that there is more than one 

7 
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person contributing to the current session in view. Tracks are preferably sorted and 
color-coded in the arrangement window, according to user. 

Connection control component 22 is also visible on the local user's display 
screen, providing' (among other things) two windows: incoming chat and outgoing chat. 
The local user can see text scrolling by from other users at remote sequencer stations 
16, and the local user at local sequencer station 14 is able to type messages to the 
other users. 

In response to a command from a remote user, a new track may appear on the 
local user's screen, and specific musical parts begin to appear in it. If the local user 
clicks "play" on his display screen, music comes through speakers at the local 
sequencer station. In other words, while the local user has been working on his tracks, 
other remote users have been making their own contributions. 

As the local user works, he "chats" with other users via connection control, 
component 22, and receives remote users' changes to their tracks as they broadcast, oi 
"post," them. The local user can also share his efforts, by recording new material and 
making changes. When ready, the local user clicks a "Post" button of client application 
component 20 on his display screen, and all remote users in the virtual studio can hear 
what the local user is hearing - live. 

As shown in Fig. 1 , local sequencer station 1 4 also includes a services 
component 24 which provides services to enable local sequencer station 14 to share 

- sequencje-data-with-remote-sequencer^^ 

including server communications and local data management. This sharing is 

8 
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accomplished by encapsulating units of sequence data into broadcast data units for 

transmission to server 12. 

Although server 12 is shown and discussed herein as a single sewer, those 

one or more individual servers. For example, it may be desirable In certain applications 
to provide one server responsible for management of broadcast dateunlts and a 
separate server responsible for other server functions, such as permissions 
management and chat administration.- 

F,g. 2 shows the subsystems of services component 24, inciuding first interface 
module 26, a data packaging module 28, a broadcast handler 30, a server 
communications module 32, and a notion queue handler 34. Services component 
24 also includes a rendering module 36-and a caching module 38. Of these 
subsystems, only firs, interface module 26 is accessible to software of client application 
component 20. FW interface module 26 receives command^xlien. application, 
component 20 o, tonal sequencer station 14 and passes them to broadcast hand* 30 
and to data packaging module 28,, Date packaging module 28 responds ,o the received 
commands by encapsulating sequence date from local sequent station 14 into 

broadcast date unite fining the descriptive characteristics and time relationships of 
the sequence data. Date packaging module 28 also extracte sequence date from 
broadcast date units received from server 12 for access by client application 
component 20. 

Server communications module 32 responds to commands processed by the 

broadcast hand!, by transmitting broadcast date unite to server ,2 for distribution to a. 

9 - 
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least one remote sequencer station 16. Server communications module 32 also 
receives data available messages from server 12 and broadcast data units via server 
12 from one or more remote sequencer stations 16 and passes the received broadcast 
data units to data packaging module 28. In particular, server communications module 
receives data available messages from server 12 that a broadcast data unit (from 
remote sequencer stations 16) is available at the server. If the available broadcast data 
unit is of a non-media type, discussed in detail below, server communications module 
requests that the broadcast data unit be downloaded from server 12. . If the available 
broadcast data unit is of a media type, server communications module requests that the 
broadcast data unit be downloaded from server 12 only after receipt of a download 
command from client application component 20. 

Notification queue handler 34 is coupled to server communications module 32 
and responds to receipt of data available messages from server 12 by transmitting 
notifications to first interface module 26 for access by client application component 20 
of local sequencer terminal 14. 

Typically, a user at, for example, local sequencer station 1 4 will begin a project 
by recording multimedia data. This may be accomplished through use of a microphone 
and video camera to record audio and/or visual performances in the form of source 
digital audio data and source digital audio data stored on mass memory of local 
sequencer station 14. Alternatively, source data ^l^ e ^^ e6 b y^^J M }P l _ 
— instrument coupledto local-sequencer-station-14 -and-storing the-performance-in-the 
form of MIDI data. Other types of multimedia data may be recorded. 

10 
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Once the deta is recorded, it can be represented in an "arrangement" window on 
the display screen of local sequencer station 14 by client application component 20. 
Really a sequencer program. In a well known manner, the user can select and 
combine multiple recorded tracks either in their entirety or in portions, to generate an 
arrangement Client application component 20 thus represents this arrangement In the 
form of sequence data which retains the time characteristics and descriptive 
characteristics of the recorded source data. 

When the user desires to collaborate with other users at remote sequencer • 
stations 16. he accesses connection control component 22. The user provides 
commands to connection control component 22 to execute a log-in procedure in which 
connection control component 22 establishes a connection via services component 24 
through trie Interne. 18 to server 12. Using well known techniques of log-in registration 
via passwords, the user can either log in to an existing virtual studio on server 12 or 
estabfeh a new virtual studio. Virtual studios on sewer 12 contain broadcast data-untts 
generated by sequencer stations in the form of projects cdntalninfl an*ngements.as 

set forth in detail below. , 

A method consistent with the present invention will now be described. The 
method provides sharing of sequence data between local sequencer station 14 and at 
.east one remote sequencer station 16 over network 18 via server 12. As noted above, 
the sequence data represents audiovisual occurrences each having a descriptive 
characteristics and time characteristics. 

When the user desires to contribute sequence data generated on his sequence 
station to either a new or listing virtual studio, the user activates a POST button on his 

11 
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screen which causes client application component 20 to send commands to service 
component 24. A method consistent with the present invention includes receiving 
commands at services component 24 via client application component 20 from a user 
at local sequencer station 14. Broadcast handler 30 of service component 24 responds 
to the received commands by encapsulating sequence data from local sequencer 
station 14 into broadcast data units retaining the descriptive characteristics and time 
relationships of the sequence data. Broadcast handler 30 processes received 
commands by transmitting broadcast data units to server 12 via server communications 
module 32 for distribution to remote sequencer stations 16. Server communication 
module 32 receives data available messages from server 12 and transmits notifications 
to the client application component 20. Server communication module 32 responds to 
commands received from client application component 20 to request download of 
broadcast data units from the server 12. Server communication module 32 receives 
broadcast data units via the server from the at least one remote sequencer station. 
Data packaging module 28 then extracts sequence data from broadcast data units 
received from server 12 for access by client application component 20. 

When a user is working on a project in a virtual studio, he is actually 
manipulating sets of broadcast data managed and persisted by server 1 2, In the 
preferred embodiment,, services Component 24 uses an object-oriented data model 
managed and manipulated by data packaging module 28 to represent the broadcast 

.data -Byj'ft'"g hrnarirast data units in th e form of objects created by serv ices 

component 24 from sequence data, users can define a hierarchy and map 
interdependencies of sequence data in the project. 

.12 
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Fig. 3 shows the high level containment hierarchy for objects constituting 
broadcast data unfts in me preferred embodiment. Each broadcast object provides a 
set of interfaces to manipulate the object's attributes and perform operations on the 
object. Copies of all broadcast objects are held by services component 24. 
Broadcast objects are created in one of two ways: 
. Creating objects locally and broadcasting then*, sewer 12. Client 
applicafion component 20 creates broadcast objects local* by calling Create methods 
(set forth in detail in the Appendix) on other objects in the hierarchy. 

. Receiving a new broadcast object from server 12. When a broadcast 

object Is broadcast to server 12, it is added to a Pn*« Database on the server and 
broadcast to all remote sequence stations connected to the project 

Services component 24 uses a notification system of notation queue handler 
34 to communicate w«h Cent applicafion component 20. Nofificafions allow setvices 
^a^i^-WW. changes in the stetes ^broadcast--- 

objects. 

Client appBcaton 20 is often in a state in which fine date it is using should not be 
cnanged. For example, If a sequencer application is in the middle of playing bacx a 
sequence of date from a file. » may be important that it finish playback before the data 
- ischanged. .n order to ensure that Ha does no, happen, nofificafion queue hand.er.34 
„, services component 24 only sends notions in response to a request by cfien. 
eppiication component 20. allowing client application component 20 «o handle .he 
notification when it Is safe or convenient to do so. 
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At the top of the broadcast object model of data packaging module 28 is Project, 
Fig. 3. A Project object is the root of the broadcast object model and provides the 
primary context for collaboration, containing all objects that must be globally accessed 
from within the project. The Project object can be thought of as containing sets or 
5 "pools" of objects that act as compositional elements within the project object. The 
Arrangement object is the highest level compositional element in the Object Model. 

As shown in Fig. 4, an Arrangement object is a collection of Track objects. This 
grouping of track objects serves two purposes: 

1 . It allows the Arrangement to define the compositional context of the 

10 tracks. 

2. It allows the Arrangement to set the time context for these tracks. 
Track objects, Fig. 5, are the highest level containers for Event objects, setting 

their time context All Event objects in a Track object start at a time relative to the 
beginning of a track object. Track objects are also the most commonly used units of 

1 5 ownership in a collaborative setting. Data packaging module 28 thus encapsulates the 
sequence data info broadcast data units, or objects, including an arrangement object 
establishing a time reference, and at least one track object having a track time 
reference corresponding to the arrangement time reference. Each Track object has at 
least one associated event object representing an audiovisual occurrence at a specified 

20 time with respect to the associated track time reference. 

!rhe-sequence-dataproduced-by-client-application-component-20-of-local_ 

sequencer station 14 includes multimedia data source data units derived from recorded 
data. Typically this recorded data will be MIDI data, digital audio data, or digital video 



PCT/USOO/25977 

WO 01/22398 

date, though any type of data can be recorded and stored. These multimedia data 
souroe date units used in the Project are represented by a type of broadcast data units 
Known as Asset objects. As Fig. 6 shows. an Asset object has an associated set of 
Rendering objects. Asset objects use these Rendering objects to represent different 
•views" of a particular piece of media, thus Asset and Rendering objects are designated 
as media broadcast data units. All broadcast data units other than Asset and 
Rendering objects are of a type designated as non-media broadcast data unite. 

Each Asset object has a special Rendering object that represents the original 
source recording of the data. Because digital media data is often very large, this 
origina, source date may never be distributed across the network. Instead, compressed 
versions of the date will be sent. These compressed versions are represented as 
alternate Rendering objects of the Asset object 

By defining high-level methods, (set forth in detail in the Appendix), for setting 
and manipulattng teese Rindering- objects. Asset objects provide a means of managing 
various versions of source data, grouping them es a common compositional element. 
Date packaging module 28 thus encapsulates the multimedia spume objects into at 
least one type of asset rendering broadcast object each asset rendering object type 
specifying a version of multimedia date source date exhibiting a different degree of 
data compression. 

The sequence data unite produced by client application component 20 of local 
sequencer station 14 include clip data . units each representing a specified portion of a 
multimedia date source data unit. Date packaging module 28 encapsulates these 
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sequence data units as Clip objects, which are used to reference a section of an Asset 
object, as shown in Fig. 7. The primary purpose of the Clip object is to define the 
portions of the Asset object that are compositionally relevant. For example, an Asset 
object representing a drum part could be twenty bars long. A Clip object could be used 
to reference four-bar sections of the original recording. These Clip objects could then 
be used as loops or to rearrange the drum part. 

Clip objects are incorporated into arrangement objects using Clip Event objects. 
As shown in Fig. 8, a Clip Event object is a type of event object that is used to reference 
a Clip object. That is, data packaging module 28 encapsulates sequence data units 
into broadcast data units known as Clip Event objects each representing a specified 
portion of a multimedia data source data unit beginning at a specified time with respect 
to an associated track time reference. 

At first glance, having two levels of indirection to Asset objects may. seem to be 
overly complicated. The need for it is simple, however: compositions are often built by 
reusing common elements. These elements typically relate loan Asset object, but do 
not use the entire recorded data of the Asset object. Thus, it is Clip objects that identify 
the portions of Asset objects that are actually of interest within the composition. 

Though there are many applications that could successfully operate using only 
Arrangement/Track, and Clip Event objects, many types of client application 
components also require that compositional elements be nested. 

Eor-example,-a-drum.partxould-he-arranged^^ 

each track represents an individual drum (i.e., snare, bass drum, and cymbal). Though 
a compos- r may build up a drum part using these individual drum tracks, he thinks of 

16 
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the whole drum part as a single compositional element and will-alter he is done 
editing-manipulate ft complete drum arrangement as a single part. Many dient 
application components create folders for these tracks, a nested part that can then be 
edited and arranged as a single unit 

. in order to allow this nesting, the broadcast object hierarchy of data packaging 
module 26 has a special kind of Event object called a Scope Event object. Fig. 9. 

A Scope Event object is a type of Event object that contains one. or more 
Timeline objects. These Trmeline objecfc in turn contain further events, providing a 
nesting mechanism. Scope Event objects are thus very similar to Arrangement objects: 
the Scope Event object sets the start time (the time context) for all of fte Trmeline __ 
objects it contains. 

Trmeiine objects are very similar to Track objects, so that Event objects that 
these Timeline objects contain are all relative to the start time of the Scope Event 
object: 

Event-data objects each having a Scope Even, time reference estobflehed*. a specific 
*. with respect to an associated track time reference. Each Scope Event object 
includes a. least one Timeline Even, object, each TimeHne Event object having a 
Trmellne Even. time reference established at a specffic time wift respect to .he 
assorted scope even, time reference and Including at least one Even, object 
presenting an audiovisual occurrence at a specified time with respec .o the 
associated timeline event time reference. 

A Project objec. contains zero or mora Custom Objects. F,g. 10. Custom 
Objects provide a mechanism for containing any generic da.a .ha. client application 
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component 20 might want to use. Custom Objects are managed by the Project object 

and can be referenced any number of times by other broadcast objects. 

The broadcast object model implemented by data packaging module 28 contains 

two special objects: rocket object and extendable. All broadcast objects derive from 

these classes, as shown in Fig. 11. 

Rocket object contains methods and attributes that are common to all objects in 

the hierarchy: (For example, all objects in the hierarchy have a Name attribute.) 

Extendable objects are objects that can be extended by client.application 
component 20. As shown in Fig.1 1 , these objects constitute standard broadcast data 
units which express the hierarchy of sequence data, including Project. Arrangement, 
Track, Event, Timeline, Asset, and Rendering objects. The extendable nature of these 
standard broadcast data units allows 3 rt party developers to create specialized types of 
broadcast data units for their own use. For example, client application component 20 
could allow data packaging module 28 to implement a specialized object called a 
MixTrack object, which includes all attributes of a standard Track object and also 
includes additional attributes. Client application component 20 establishes the 
MixTrack object by extending the Track object via the Track class. 

As stated above. Extendable broadcast data units can be extended to support 
specialized data types. Many client application components 20 will, however, be using 
common data types to build compositions. Music sequencer applications, for example, 

~ WalmosTalways be uslng-DigltalAudio-and-MIDI-data-types.- 
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Connection control component 22 offers the user access to communication and 
navigation services within the virtual studio environment. Specifically, connection 
control component 22 responds to commands received from the user at local 
sequencer station 14 to. establish access via 12 server to a predetermined subset of 
broadcast data units stored on server 12. Connection control component 22,contains 

these major modules: 

1. A log-in dialog. 

2. A pass-through interface to an external web browser providing access 
to the resource server 12. 

3. A floating chat interface. 

4. a private chat interface 

5 Audio compression codec preferences. 

6. An interface for client specific user preferences. 

The log-in dialog permrts the user to either create a new account at server 12 or 
,og-in to various virtual studios maintained on server 12 by entering a previously 
registered user name and password. Connection control component 22 connects the 
user to server 12 and establishes a web browser connection. 

Once a connection is established, the user can search through available virtual 
studios on server 12, specify a studio to "enter," and exchange chat messages with 
other users from remote sequence stations 16 through a chat window. 

In particular, connection control component 22 passes commands to services 
component 24 which exchanges messages with server 12 via server communication 
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module 32. Preferably, chat messages are implemented via a Multi User Domain, 
Object Oriented (MOO) protocol. 

Server communication module 32 receives data from other modules of services 
component 24 for transmission to server 12 and also receives data from server 12 for 
processing by client application component 20 and connection control component 22. 
This communication is in the form of messages to support transactions, that is, batches 
of messages sent to and from server 12 to achieve a specific function. The functions 
performed by server communication module 32 include downloading a single object, 
downloading an object and its children, downloading media data, uploading 
broadcasted data unit to server 12, logging in to server 12 to select a studio, logging in 
to server 12 to access data, and locating a studio. 

These functions are achieved by a plurality of message types, described below. 

ACK 

This is a single acknowledgement of receipt. 

NACK 

This message is a no-acknowledge and includes an error code. 
Request single object 

Thismessage Identifies the studio, identifies the project containing the object, 
and identifies the class of the object 

Request object and children . _ _ „ 

This mess ag e identifies the studio, Identifies the project containing the object 

identifies object whose child objects and self is to be downloaded, and Identifies 
the class of object. 

20 
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Broadcast Start 

This message Identifies the studio and Identifies the project being broadcast. 



Broadcast Create 

■ This messege identifies the studio, identifies the project containing the object. 
Identifies the object being created! and contains the object's data. 

Broadcast Update 

This message Identifies the studio, identifies the project containing the object. 
Identifies the object being updeted, Identifies the dass of object being updated, 
and contains the object's data. 

Broadcast Delete 

This messege Identifies the studio. Identifies the pro|ect containing the object, 
identifies the object being deleted, and Identifies the dass of objed being 
updated. 



Broadcast Finish 

This message Identifies the studio, and Identifies the project being broadcast 
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Cancel transaction 

This message cancels the current transaction. 

Start object download 

This messege identifies the objed being downloaded in this message. Identifies 
,he dass of object, identifies the parent of the objed. and contains the object's 
data. 

• Single object downloaded 

This message Identifies the object being downloaded. Identifies the dass of the 

objed. and contains the objed data. 
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Request media d wnload 

This message Identifies the studio, identifies the project containing the object, 
identifies the rendering object associated with the media to be downloaded, and 
identifies the dass of object (always Rendering). 

Broadcast Media 

This message identifies the studio, identifies the project containing the object, 
identifies the Media object to be uploaded, identifies the dass of object .(always 
Media), identifies the Media's Rendering parent object, anleontalns Media data. 

Media Download 

This message Identifies the rendering object associated with the media to be 
downloaded, Identifies the class of object (always Rendering), and contains the 
media data 

Request Tlmestamp 

This message requests a timestamp. 



Response Tlmestamp 

This message contains a timestamp in the format YYYYMMDDHHMMSSMMM 
(Year. Month. Day of Month. Hour. Minute. Second, Milliseconds). • 

Request Login 

This message Identifies the name of user attempting to Login and provides an 
MD5 digest for security. 

Response SSS Login 

This message indicates if a user has a registered 'Pro' version; and provides a 
Session token, a URL for the server Web site, a port for data server, and the 
address of-OiB data-server -~- 
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Request Studio L catl n 

This message identifies the studio whose location is being requested and 

me community and studio names. 

Response StudloLocatlon 

" This message identifies the studio, the port for the MOO, and the address of the 

MOO. 

Request single object 

' This message identifies the studio, identifies pmject containing the object, 
identifies object to be downloaded, and Identifies th« dass of ob|ect, 

Finish object download 

the class of obiect, and identifies the parent of ob)ecL 

Client application componen.20 gains access to services component 24 fiuough 
a set of in.er.ace dasses defining firs, interface module 26 and contested in a Cass 
Horary, in thepmferred embodimem.heseWs arc implemented in straight*™,*, 

• enorial knowledae of COM or other inter-process 
cross-platform C++ and require no special knowieoge 

communications technology. 

A sequencer manufacterer integrates a diem application component 20 .o 
service component 24 by linking .he dass .ibrary to source code of dien. applicafion 
component 20 in a well-Known manner, using for example, .sua, C~ for Windows 
application or Me,rowo*s Codewanier (Pro Release 4) for Madntosh applications. 
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Exception handling is enabled by: 

Adding Initialization and Termination entry points to client application 

component 20 ( ^initialize and .terminate), 

Adding "MSL RuntimePPO+.DLL" to client application component 20, 
5 and 

Add "MSL AppRuntime.Lib 0 to client application component 20 
- Once these paths are specified, headers of services component 24 simply 

are included in source files as heeded. 
A detailed description of the classes of the class library necessary to implement 
10 a system consistent with the present invention is set forth in the Appendix. 

To client application component 24, the most fundamental class in the first 
interface module 26 is erwservices. It provides methods for performing the following 
functions: 

Initializing Services component 24. 
15 • Shutting down Services component 24. 

Receiving Notifications from Services component 24. 
Creating Project objects. 

Handling the broadcast of objects to Server 12 through services 
component 24. 

20 • Querying for other broadcast object interfaces. 
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Each implementation that uses services component 24 is unique. Therefore the 
first step is to create a services component 24 class. To do this, a developer simply 
creates a new class derived from «*tsorvi« B : 

ClaSS CMyRlctServices : public crKtServices 
I 

public. CMyRlctServices ( ) ; 

virtual -CMyRlctServices ( ) t 
etc .. 

An application connects to Services component 24 by creating an instance of its 

CRlctServices claSS. and Calling CRktServices: initialize () : 
try 

1 cHyRoclcetServices -pMyRocketServices - new CHyRoclce t Services; 

pMyRocJcetServicea->Initialize ( ) ; 
catch ( CRrktExceptionfi e) 
* // initialize Failed 



) 

CRktServicea : : Initial!" 



o automatically performs all operations necessary to initiate 
communication withsem^ applicatonxomponent 20. 

Client application component 20disconnects Irom Se^ices component 24 . by 
deleting the aucts«rvicas instance : 

// if a Services component 2 4 Class was created/ delete it 
if (m_pR*tServices !- NULL) 

1 delete m pRktServices ; 

m_pRJctServices - NULL; 

) 



Services component 24 will automatically download only those custom date objects thet 
have been registered by the client application, provides en interface for 

doing this: 
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try 

// Register Cor our types of custom d j^f • 
m P RktService3->RegisterCustomDataType( C UST^DATATYPEID1 / 
m~pR*tServices->RegisterCustomDataType( C0STOMDATATYPEID2 ); 

r 

catch { CrktExceptionfi e) 
I 

// initialize Failed 



Like ouctservic, all broadcast objects have corresponding cm interface 
implementation classes in first interface module 26. . It is through these cwct interface 
15 classes that broadcast objects are created and manipulated. 

Broadcast objects are created in one of two ways: 

• Creating objects locally and broadcasting them to the Server. 

• Receiving a new objects from the server. 

r 

There is a three-step process to creating objects locally: 
20 1 . Client application component creates broadcast objects by calling the 

corresponding creat. o methods on their container object. 

2. Client application component calls createMctmtereaeaoto get an interface to 
that object. 

3. Client application component calls cRkts«rvie.B. :Broadc«Bto to update the 
25 server with these new objects. 

Broadcast objects have create i> methods for every type of object they contain. 
These create o methods create the broadcast object in services component 24 and 
return the ID of the object. 
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For example, d*——. has methods for creating a Project. The following code 



would create a Project using this method: 

SIS? S K^^'try-catch for possible error conditions 



' // attempt to create project 

. '^tSe^^ 
CRktServtces:sinstance()->CreatePrqject{) )j 

■ // user created, set default name 
pProject->SetName( »Nav Project") . 

) // try 

catch ( CRktExceptiont e ) 

' delete pProjectf 

e . Report WctBrror 0 . 
return false; 

' To create e Track, client application component 20 calls the mU~* method 
of the Arrangementobject. Each parent broadcast object has methods create tts 
specific types of child broadcast objects. 

H is not necessary (nor desirable) to call «^c. .« immediately after 
creating new broadcast objects. Broadcasting is preferrably tnggered from the user- 
interface of client application component 20. (Wren the user * a -Broadcast" buKon. 
for instance): 

Because services component 24 keepstrack of and manages all changed 
broadcast objects, client applicafion component 20 can take advantage of the data 
management of services component 24 while allowing users to choose when to share 

Note that (unlike data model Interface objects are not created 

directly. The must be created through the creetion methods or the parent object. 
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Client application component 20 can get cm interface objects at any time. The 

objects are not deleted from data packaging module 28 until the Removed method has 

successfully completed. 

Client application component 20 accesses a broadcast object as follows: 

// Get an interface to the new project end 
// eet name. 

C&ktPtr < CRktP reject > ptfyProject * 

GHyRktServieoot xXnetaneeO ->CraateWctProjectXnter£ece (Project) i 
HyPreject->Bettfame < stProjMaae) ; 

> // try 

catch ( CRktBxeeptionfc e ) 

{ 

e .Report Me tBrror () i 

' The cRktPtr<> template class is used to declare auto-pointer objects. This is 
useful for declaring interface objects which are destroyed automatically, when the 
otttptr goes out of scope. 

To modify the attributes of a broadcast object, client application component 20 
calls the access methods defined for the attribute on the corresponding CRkt interface 
class: 

// Change the name of my project - 
pRktObj->SetName< M My Project* ); 

Each broadcast object has an associated Editor that is the only user allowed to 
make modifications to that object. When an object is created, the user that creates the 
object will become the Editor by default. 

Before sen/ices component 24 modifies an object it checks to make sure that the 
current user is the Editor for the object If the user-does not have permission to modify 
the object or the object is currently being broadcast to the server, the operation will fail. 
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Once created, client application component 20 is responsible for deleting the 
interface object: 

delete pTrack; . ... . . , 

Dieting », interface classes should not be confused «ith , pemovng the object 
from the data model. To temove an object from the data model, you call the object's 
bbbovo method is called: 

P Track->Removel); // mmove Irem the datB.model 

Interface objects are "reference-counted." Although calling — o will 
effectively remove the object from the data model, it will not de-allocate the interface to 
it. The code for properly removing an object from the data model is: 

CRktTrack* pTr ack ; 

Liota pxrack; * delete the .ntarface object 

or using the cwctptr Template: 



CHJctPtr < OtrktTrack > pTraeki 
// create Intertac* -. 

^reS"^ 2— U«Ur b. deleted -hen it 
// goaa out of ocope 

Like the create process, objects are not deleted globally until the 
cm8.rvic..,.Bro.dc«t() method is called. 

If the user does not have permission to modify the object or a broadcast is in 
progress, the operation will fail, throwing an exception. 

Broadcast objects are not sent and committed to Server 12 until the 
cttt s.^e..,. B ~.de.. t o interface method is called. This allows users to make changes 
.ocally before committing them to the server and other users. The broadcast process is 
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an asynchronous operation. This allows client application component 20 to proceed 
even as data is being uploaded. 

To ensure that its database remains consistent during the broadcast procedure, 
services component 24 does not allow any objects to be modified while a broadcast is 
in progress. When all changed objects have been sent to the server, an 
o a Bro.de«.te«piet6 notification will be sent to the client application. 

Client application component 20 can revert any changes it has made to the 
object model before committing them to server 12 by calling auctse*vice 8 . .r»iu»gm> . 
When this operation is called, the objects revert back to the state they were in before 
the last broadcast. (This operation does not apply to media data.) 
noiibaoko is a synchronous method. 

Client application component 20 can cancel an in-progress broadcast by calling 
cr fct8 .rvtc..„c»e^Bro«ic«to . This process reverts all objects to the state they are in on 
the broadcasting machine. This includes all objects that were broadcast before 

CancolBroadcastO WSS Called. 

j cancawroadcMto is a synchronous method. 

Notifications are the primary mechanism that services component 24 uses to 
communicate with client application component 20. When a broadcast data unit is 
broadcast to server 12, it is added to the Project Database on server 12 and a data 
available message js rebroadcast to all other sequencer stations connected to the 

Hpleje^SeTvicwoo™ 
for their associated client application component 20. For non-media broadcast data 
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units, the other sequencer stations also immediately request download of the available 
broadcast data units; for media broadcast data units, a command from the associated 
client application component 20 must be received before a request for download of the 
available broadcast data units is generated. 
5 . Upon receipt of a new broadcast data unit, services component 24 generates a 

notification for client application component 20. For example, if an Asset object were 
received, the o n cr..^-.tc^i.t.<, notification would be generated. 

All Notifications are handled by the ou^o.. instance and are implemented 
as virtual functions of the cRktsenricee object. 
10 To handle a Notification, client application component 20 overrides the 

corresponding virtual function in its cRxts.rvic class. For example: 
class CMyRktServices :• public CRktSeryices 
1= //Overriding to handle OnCreateAssetComptete NotiRcatoiw 

'° virtual void OnCreatehssetComplete ( 

* When client application component 20 receives notations via notification 
queue handler 28, these overridden methods will be called: 
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WctNestType 

CMyWtt8»rvic«»::OnCr«ataAns»tStmrt < 
25 const RktObJectldtypeS 

^ft^tabiectldType* rParentObiectld ) 
( 

try 

30 i 

// Add this Arrangement to My Project 
• " ' m ^r 0 ££^>™^l« '« r P ar.nt«,JectId-rObi.e«d„ , // trv 
catch ( CRktExceptiont e ) 

^ ' e.ReportRktErrorOJ 

return BOCKBI_Qt7BUB_nO_MK8Il 

) 
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Sequencers ar often in states in which the data they are using should not be 
changed. For example, if client application component 20 is in the middle of playing 
back a sequence of data from a file, it may be important that it finish playback before 
the data is changed. 

In order to ensure data integrity, all notification transmissions are requested 
client application component 20, allowing it to handle the notification from within its own 
thread. When a notification is available, a message is sent to client application 
component 20. 

On sequencer stations using. Windows, this notification comes in the form of a 
Window Message. In order to receive the notification, the callback window and 
notification message must be set. This is done using the 
CRktServices: :SetDataNotificationHandlor() method: 

// Dafine > maasage tor notification from Service* coaponent 24. 
•define RKTKS8_H0tXFICaTI(m_PBMDIH0 < MK.APP ♦ OxlOO ) 

// Mow Set the window to be notified of Rocket Bvente atjrRfctServicoe.xInetanceO - 

>SetDataHotiflcationHandler { »_Mtnd, » 

RKTMSG _HOTIFIC*TIOH_PBHDINO) / 

This window will then receive the rxtmsg_notification_pending message 
whenever there are notifications present on the event queue of queue handler 
module 34. 

Client application component 20 would then call 
CRktServicea: :ProcessNextDataNotication() to instruct services component 

24 to send notifications for the next pending d ata notificatio n: J 

// Data available for Bocket Services. Request Notifieatiod. 
afx_msg otainFrame; :OnPendingDae»Notification(LPARMI l.HPARMt w> 

CMySktServicesi tXnetaneeO ->Proce»aMe«tDataHotif leatlonO » 
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irocesBNextDataNotif ication ( ) causes services component 24 to remove the 
notification from the queue and call the corresponding notification handler, which client 
application component 20 has overridden in its implementation of a*t,.~ie... 

On a Macintosh sequencer station, client application component 20 places a call 

tO CrfctServic«8 ; : - 

D oHotiti«.tion. U in th.tr idle loop, — - 
anuat^ptificatlonxvilableo d ' notification 

If This method called when data available on the event nocnx 

void^iktserviceo : :OnDataNotif IcationXvailableO 



I 

try 



) 

) 



Proco saNaxtDft taHo tif icati on ( ) ; 
catch < CWctLogicBxcaption © ) 
e . RnportWctError ( ) ; 



As described in the Windows section above, P roc.«H„t^t- 1 .oti«ic.tiona;instructs 
services component 24 to remove the notification from the queue and call the 
corresponding notification handler which ^w^a^'axio^T^ ■ - 
overridden in its implementation of aoets.rvie.8. 

Because notifications are handled only when client application component 20 
requests them, notification queue handler of services component 24 uses a "smart 
queue" system to process pending notifications. The purpose of this is two-fold: 

1 . To remove redundant messages. 

2. To ensure that when an object is deleted, all child object messages are 
removed from the queue. 
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This process helps ensure data integrity in the event that notifications come in 
before client application component 20 has processed all notifications on the queue. • 

The system of Fig.1 provides the capability to select whether or not to send 
notifications for objects contained within other objects. If a value of Roa«T_gow«_Do_HB8T 
5 is returned from a start notification then all notifications for objects contained by the 
object will be sent. If mckbt.qowib.do.mot.iibst is returned, then no notifications will be sent 
for contained objects. The croate<T>con>iet. notification will indicate that the object and 
all child objects have been created. 

For example if client application component 20 wanted to be sure to never 
1 0 receive notifications for any Events contained by Tracks, it would override the 
oncre.t.projectst.rto method and have it return rocket_qokob_do_hot_ksst: 

KktHestType 

CMylUctServices: : OnCreateProjeetStart ( 

const WctObjeetldTypet rObjactld, 
15 const KktObjectldlypeS rParontOtojectrd I 

// don't aend ma notifications tor 

// anything contained by thio project. 

return ROCKST QUEUS DO_MOT_KKST; 

20 > 

And in the ewKMWii. notification parse the objects contained by the track: 

void 

CMyRJctSarvicea: .OnCreataPrejactC 
25 omploto ( 

conat WttQbjectXdTypet 
objoctld, 

const RktObjoctidTypot 
parentObjactZd ) 
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In the pretend embodiment, pre defined br oadcast objects are used wherever 
possible. By doing this, a common interchange standard is supported. Most client 
application components 20 will be able to make extensive use of the predefined objects 
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in the broadcast object Model. Thereat times, however, when a olier* application 

component 20 will have to tailor objects to its own use. 

The described system provides two primary methods for creating custom and 

extended object If dent application component 20 has-an objectwhich is a .variation 
of one of the objects in the broadcast object model, ft can choose to extend the 
broadcast object. This permits retenflon of all of the attributes, methods and 
containment of the broadcast object. wMe tailoring ft to a specific use. Forexample. if 
client application component 20 has a type of Track which holds Mix information, ft can 
extend the Track Object to hold attributes which apply to the Mix Track .mplementation. 
All predefined broadcast object data types in the present invention (audio. MIDI, M.DI 
Drum, Tempo) are implemented using this extension mechanism. 

The first step inextending a broadcast object is to define a -globally unique 

RktExtendedpataldType: 

,/ a g lobally-uni,ue £*. ^^^^^^gU^ 

This ID is used to mark the data type of the object. It allows services component 20 to 
know what type of data broadcast object contains. The next step is to create an 
attribute structure to hold the extended attribute data for the object 

etruet wyTracleAttributee 

{ c*yTracWlttritoutee<)i 

Int32Type -nHyfluanti.e, // «V extended dete 

)> 

II Simple way to Initialize defaults for your attributes is 
// to use the constructor for the struct 
CKyTrecJcAttributee: , WyTrackAttrlbutee 0 
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a nXyQuaatise e kMyDef aultQuaatixe; 

} 



To initializ an extended object, client application component 20 sets the data 
type Id, the data size, and the data: 



// set my attributes 

CHyTraekAt tributes myTraekAt tributes i 
oyTrackAt tributes ;»_n)tyQuantixe » 16; 



try 

10 . { 



// Set the extended data type 

p?rack->SetDataType< mf w RXTBMD8P^TRAat - _XTTR - .lD )l 



// Set the data (and length) 

1 5 lnt32Type nsiie - eiieof (myTrackattributes) i 

Track- >SetData ( fcmyTracxAttributas, trnSise) i 

) 

eaten ( CRktException e ) 

{ 

20 • . ReportRktBrror {) t 

) 

When a notification is received for an object of the extended type, it is assumed 
to have been initialized. Client application component 20 simply requests the attribute 
25 structure from the cRkt interface and use its values as necessary. 

// Check the data type, to see if we understand it. 
RktRxtandedDataldType dataType ■ 
pfraek - >OatDat aType < > I 

30 // if this is a MIDI track - 

if ( dataType « CLSH> ROCKET KIDX_TRXCX_ATTR J 

{ 

// Create a Midi struct 
- CHyTrackAttrlbutee ayTraekAttributea t 
35 // cat the Data. Upon return, nSiae is set to the actual 

// size of the data. ■ 

XatJSType nfliae » siseof < OiyTrackAt tributes )i 
pTrack->OetData -{ fcmyTrackAttributes, nSiae >i 

40 // Access struct members... 

DoSo&athiagtfith( nyTrackAttributee )> 



Custom Objects are used to create proprietary objects which do not directly map 
45 to objects in the broadcast object model of data packaging module 28. A Custom Data 
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Object is a broadcast object which holds arbitrary binary data. Custom Data Objects 
also have attributes which specify the type of data contained by the object so that 
applications can identify the Data object. Servic.s component 24 does provide all of 
the normal services associated with broadcast objects - Creation. Deletion, Modification 
methods and Notifications - for Custom Data Descriptors. 

The first step to creating a new type of Custom Data is to create a unique ID that 
signifies the data type (or class) of the object 

// a globally unique ID to identify my cuatom data object 
conat'LtCu-L-DataldTyp. «- CT «£-~"f- in 
(•rro24F40-B616-lld3-BI»7B-0060979C492B->f 

This ID must be guaranteed to be un«,ue. as this ID is used to determine the type of 
date being sent when Custom Data notations are received, .A55?^>«*» * 
define a sttucture to hold the attributes and data for the custom d^object 

Btruct CHyCuetonDataBloclc 

^ CMyCuatomDataBiock 

int m nXyCuatoaAttributaj ■ \ * — V 

>l " 

^.....c..™^,, oan be called to oreate a new custom object, set the data 
type of the Data Descriptor object, and set the attribute structure on the object: 



try 

{ 



- - — " ob3ect 

WctObjectXdType nyCustomObjectld - 
pProject->CreateCuoto»Objacc ( 

// Get an interface to it _ , _ _ 

( BjyCuotoiaObjectld )y 
// Create my custom data block and fill it in. . . 
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OtyCustoaDataBlock nyCuatoaData; 



// Set the custom data type 

pCuatomObject«>SetDataType( MY_CUSTOX_OBJECT_ID )j 

// Attach the extended data to the object (set data and size) 
Xat32Type ©Site - eiseof< CHyCufltoapataBloek )t 
pCufltoaObject->SetD*ta( fcmyCuatomData, nSisa ); 

} // try 

catch' ( CRktBxeaption e ) 

{ 

e.ReportRktBrrorO r 

} 



When client application component 20 receives the notification for the object, it 
simply checks the data type and handles it as necessary: 

// to aeeeaa an existing Custom Data Object i 

try ^ 

// Assume we start with the ID of the object. . . 

// Get an Interface to it 
CRktPtrx CWctCustomObject > 
pCustoaObjeet b 

»J>MyRocxet8ervice»-»CreateRktCuetoaObjectIater£ace 

( 

myCustomObjectXd ) i 

// Check the data type, to see if we understand it. Shouldn't 
// be necessary, since we only register for ones we understand, 
// but we'll be safe 
RktCustonDataldType idCustomr 
idCuatom r 

>; 

if ( idCustom »o CLSIDJfYJWCTOM_DATA ) 

{ 

// Create my custom data struct 
CHyCustoaDataBlock rayCustomDataj 

// Get the Data. Upon return. theSize is set to the actual 
// size of the data. 

Int32Type nSize - oiaeof ( myCuetoaData ) t 
pCusto»Objeet-»OetData{ aayCustonData, nSlze )i 

// AcceuB struct members... 
DoSomothingHitM myCuotomData )i 

. } .//—it niy„custom_data. : 

} // try 

catch ( CWctExceptionfc e ) 

{ 

e.aeportRktBrrorO j 

} 
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All of the custom data types must be registered with servioes component 24 
(during services component 24 Mafeatton). Services component 24 will only altow 
creation and reception o, custom objects which have been registered. Once registered, 
the data will be downloaded automatically. 

yyhen a user is building a musical composition, he or she arranges clips of data 
ma, reference recorded media. This recorded media is represented by an Asset object 
in ,he broadcast object model of date Paging component 32. An Asset object*-- 
Wended to represent a recorded composHional element. ..Is these Asset objecte drat 
are referenced by clips to form arrangements. 

Though each Asset object represents a single element, there can be several 

versionsof the actual ^^^ to*^ ™ ^ ™"* 1 **** 
various versions of the Asset Interna, to the Asset, each of diversions Is 
represented by a Rendering object. 

Asset date is often ve^ large and it is high,, desirable for users to broadcast 
compressed versions of Asset data: Because mis composed date wi» often be 
degraded versions of me original recording, an Asset cannot simply replace the original 
media data with the compressed data. 
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Asset objects provid a mechanism for tracking each version of the data and 
associating them with the original source data, as well as specifying which version(s) to 
broadcast to server 12. This is accomplished via Rendering objects. 

Each Asset object has a list of one or more Rendering objects, as shown in 
Fig. 6. For each Asset object, there is a Source Rendering object, that represents the 
original, bit-accurate data. Alternate Rendering objects are derived from this original 
source data. 

The data for each rendering object is only broadcast to server 12 when specified 
by client application component 20. Likewise, rendering object data is only downloaded 
from server 12 when requested by client application component 20. 

Each rendering object thus acts as a placeholder for all potential versions of an 
Asset object that the user can get, describing all attributes of the rendered data. 
Applications select which Rendering objects on server 12 to download the data for. 
based on the ratio of quality to data size. 

Rendering Objects act as File Locator Objects in the broadcast object model. In 
a sense, Assets are abstract elements; it is Rendering Objects that actually hold the 
data. 

Renderings have two methods for storing data: 
• In RAM as a data block. 

.Qji_disJcaA.a_RJe. ' 

The use of RAM or disk is largely based on the size and type of the data being 
stored. Typically, for instance, MIDI data is RAM-based, and audio data is file-based. 
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Of all objects in the broadcast object model, only Rendering objects are cached 
by cache module 36. Becaus Rendering objects are sent from server 12 on a request- 
only basis, services component 24 can check whether the Rendering object is stored 
on disk of local sequencer station 14 before sending the data-request- 

In the preferred embodiment. Asset Renderings objects are limited to three 

specific types: 

Source: Specifies the original souroe recordings Uerally represents e bit- 
accurate recreation of the originally recorded file. 

Standard: Specifies the standard rendering of the file to use. generally a 

moderate compressed version of the original source data. 

Preview: Specifies the rendering that shouU be downloaded in order to get a 

Each of the high-leve, Asset calls set forth in the ; Appendix uset ,a flag specifying 
which of me three Rendering object types is being referenced by me oa„. Typicatly me 
type of Rendering object selected will be based on the type of data confined by the 
Asset. Simple data types - such as MIDI - will no. use compression or alternative 
renderings. Mora complex data types - such as Audio or Video - use a number of 
different rendering objects to facilitate efficient usedf bandwidth. 

A first example of use of asset objects will be described using MIDI data. 
Because the amount of data is relauvely am* only the source rendering object is 
broadcast, with no compression and no alternative rendering types. 
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The sender creates a new Asset object, sets Its data, and broadcasts it to 

server 12. 

Step 1 : Create an Asset Object 

The first step for client application component 20 is to create an Asset object. 
This is done in the normal manner 

// Attempt to Create an Asset in the current Project 
MctObjectxdtyp* aseetld • p*roj*ofcs»Cre»5«As.sftO » 

Step 2: Set the Asset Data and Data Kind 

The next step is to set the data and data kind for the object. In this case, 
because the amount of data that we are sending is small, only the source data is set: 

// Set the data Eor my midi data 
pMldiAsset->BetDataKind ( DATAKIHD_ROCICKT_MIDI >» 
// Set the Midi Data 

pjliciAssot->SetSoureeMedla { pMXDXData, nMXDXDataSiae 

The setsourcnedi. o call is used to set the data on the Source rendering. The 
data kind of the data is set to »TAw»_»ocwrKtDx to signify that the data is in standard 
MIDI file format. 
Step 3: Set the Asset Flags 

The third step is to set the flags for the Asset. These flags specify which 
rendering of the asset to upload to the server 12 the next time a call to B ro.deaetc> is 
made. In this case, only the source data is required. 

// Always Broadcast MIDI 
Source 

pKidiAaaet>>Setsroadeastrlags < 

— ASSET-8H0ADCAST-80OTGB— )-» ; 

Setting the asset_broadcast_source flag specifies that the source rendenng must 
be uploaded for the object. 
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Step 4: Broadcast 

The last step is to broadcast. This is done as normal, in response to a command 
generated by the user : 

pHyRockofcServicoB- 

'"niceive an Asset, client application component 20 of local sequence stafon 
14 handles the new Asset notification and requests the asset data. Whenihe 

notation is received, the Asset object has,been created by data 
packaging module 28. Client application component 20 creates an interface to the 
Asset object and queries rts attributes and available renderings : 

const RttObJ.ctldType* rP.rentObJ.ctld ) 

^ I 

// Get an interface to the new asset 
• <3Uet*tr < CBktAaaet > pAeaat - 

^L-.at.IUctAae.tInterf.c. ( rObj-ctld ) , 



{ 



} 



{ II create one of my application- 8 MIDI aaaet equiv 

// etc. . . 

La if I detaUnd - CLSID.ROCKET.ATOIO.AESKT ) 
{ // create one of my application- a Audio asset equiv 

// etc- 

) 

latch ( CPJttException *e ) 
e.ReportRktBrrorOl 
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Data must always be requested by local sequencer station 12 for assets. This 
allows for flexibility when receiving large amounts of data. To do this client application 
component 20 simply initiates the download: 

virtual void 

Ofy WctS arvicea:sOnABB«tModi«Avai labia ( 

const RktObjectXdTypefc rAasetXd, 
const RendClaesType classification, 
const RktObjectXdTypefc rRanderingld 

{ 

try 

CRktFtr < CRktAaset > pAseet • 

CreateRktAssetXnterf ace ( rAasetXd ) ; 

// Check if the media already exists on this machine. 

// If not, download it. (Note: this isn't necessarily 

// recommended - you should download media whenever 

// it is appropriate. Your Ul might even allow downloading 

// of assets on an individual basis) . 

// Source is . always Decompressed. 
//Other renderings download compressed. 
RandStateType rendState r 

if ( classification o« AS SBT_SOURCB_RBNir CLASS ) 
rendState - ASSET J3BCOMF RES SBD_RBHD_STATB; 
else 

rendState » ASSET JZOKP RES 8 SD_RBXI)_S TATE ; 

// If the media is not already local, then download it- 

if ( l P Assat->XsMadiaLocal ( claaaif ication, rendState ) ) 

* // Note: If this media is RAM-based, the file locator 

// is ignored. 
CRktPileLocator fileLocUnuaed? 
p As set* >Downl oadxedi a 

( classification, f ileLocUnuoed ) j 

} 

catch ( CRJctException ae ) 

^ e.ReportRktSrrorO i 

} 



When the data has been successfully downloaded, the onAsaatnediaDownioadad o 
Notification will be sent. At thi^oint the data is a vailable locaHy, and client application 
component 20 calls cetDatao to get a copy of the data: 

// This notification called when data has been downloaded 
virtual void 

CMyRJctServices: :OnAssetMedia Downloaded ( 
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const RktObJecildTypei SSSSwtlbn. 
'const KKXe. raenderin^d «-t 

try 

//Find my corresponding object 
CRktPtr < CRktAsset > pAsset " 

CreateRktAssetlntertace < rAssetld ); 

// store its sixe in nSize. 

// Note: this application will be responsible tor 
// freeing the memory 

void* .pData; _ 
long nSUe; 

pAsset->GetMediaCopy ( 
asset source rekdjclass, 
asseOecomprbssbd.remd^statb, ■ 

tpData, 
nSize ); 

* catch ( CWctException &e ) 

< 

e.ReportRktErrorO; 

) 

,„ a second example, an audio data Assails Seated. Client application 
exponent 20 sets the audio data and a compressed preview rendering is generated 
automatically by services component 24. . - — ---r ; .r 

,n this scenario the dab size is quite large, so the data is stored In a file. 

The sender follows many of the steps in the simple MIDI case above. This time, 
however, the data is stored in a file and a different broadcast flag used: 

// Get an interface to the new asset. 

CR £Eserv^s?t?ns^ 
( assetld ) i 

'!, X?2S^X32S£?~ **■ Justthe 

preview 

CRIctFileLocator MleLocator; 

// Set the I lleLocator here (bring up a dialog or use a 
// p!thna»e. Or use an FSSpec on). 

p S»set->SetSourceMedia( * f ileLocator-l ; 
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// Set the flags so that only a preview is uploaded. 
// We did not generate the preview rendering ourselves, 
// so we will need to call 

// CRktServices: jRenderforBroadcast 0 before calling 

// Broadcast (). This will generate any not-previously 

// created renderings which are specified to be broadcast. 

pAsset->SetBroadcastFlags ( 
ASSET^BROAPCAST_PREVlBW ) ; 

// Hake sure all renderings are created 
pMyRocket services «->RenderForBroadcast O ; 

// and Broadcast 
pHyRocketServices->Broadcast() ; 

Because asset_broadcast_prevxew was specified, services component 24 
will automatically generate the preview rendering from the specified source rendering 
and flag it for upload when CRocketServices : :RenderForBroadcast () is called. 

Alternatively, the preview could be generated by calling 

CRktAeoet: s CoopreBsMediaO explicitly: 

// compress the asset (true means synchronous) 
pAsset->CompressMedia ( 
ASSErr_PREVTEW_REND_CLASS # 7 

true )/ 

In this example asset_brom>cast^source was not set. This means that the 
Source Rendering has not been tagged for upload and will not be uploaded to server 
12. 

The source rendering could be added to uploaded later by calling: 

pAsset->SetBroadcastrlags „„„,._, . , 

. . ' ( ASSET BROADCAST SOURCE I ASSBT_BROA0CAST_PREVIE» ) I 
pMyRocketservices->Broadcast () ; 

When an Asset is created and broadcast by a remote sequencer station 16, 
notific^on queuel^ndrer 28 generates an onCreateAseetCoaipiete ( ) notification. 
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Client application component then queries for the Asset object, generally via a lookup 
by ID within its own data model: 

// fir>A matching asset in my data model, 
ciyts^t- Posset - FindMyAsset ( IdAsset , ; 

As above, the data would be requested: 

CRktFileLocator locDownloadDir ; 

/ / On Windows... _ ■ V( j . \ \ M yDownloads\\" .); 
locDownloadDir. SetPatM c^\> y fal - use an FSSpe c) 

^locDownloadDir ); 

The d*tx...t , = Dc-nlo.dM.dia 0 specffies the classification of the 
rendering data to download and the directory to which the downloaded file should be 
written. 

When the data has been successfully downloaded, the 
is available, but it needs to be decompressed: 

// this notification called -hen data has been downloaded 
. ^ocLrse d rvice S ::OnAasetHediaDownloaded 1 
Y const RlttObjectldTypet tAssetld, ■ 
const RendClassType classification, 
const WctObjectldTypet rRendenngld 



I 

try 

1 // Get an interface to the asset 

CRlctPtr < CRktAsset > pAsset - 
CreateRktAssetlnterface { rAssetld >, 

// and get set the data for the * s f". 
P ftsset->Decot»pressAenderingl classification, false ).. 

catch ( CRktException te ) 
1 e.ReportRktEr'rorO; 
) 
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When the data has been successfully decompressed, the 
OnAssetDataDecomprese d() notification will be sent: 

// This notification called when data decompression complete 
virtual void 

5 CHyRktServices; rOnAssetMediaDecompressed { 

const RJctObjectldTypes rAssetld, 
const RendClassType classification, 
const RktObjectIdType..rRenderingld ) 

10 try 

• CreateRktAssetlnterface ( rAssetld ); 

// Get the Audio data for this asset to a Tile. 
15 CRktFileLocator locDecompressedFile - 

pMyAsset->GetMedia 
(classification, 
ASSET DECOMPRESSEDJlEND_STATE )f 

// Now import the file specified by locDecompressedFile 
20 // "into Application-. 

catch ( CRktException &e ) 
t 

e.ReportRktErrorU; 

25 ) 

•/ 

Services component 24 keeps track of what files it has written to disk client 
application component 20 can then check these files to determine what files need to be 
downloaded during a data request Files that are. already available need not be 
30 downloaded: Calls to isMediaLocalO indicate if media has been downloaded 

already. 

Services component 24 uses Data Locator files to track and cache data for 
Rendering objects. Each data locator file is identified by the ID of the rendering it 
corresponds to, the time of the last modification of the rendering, and a prefix indicating 
35 whether the cached data is preprocesse^(compressed) or post-processed 
(decompressed ). 
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For file-based rendering objects, fries are written in looations specified by the 
client application. This allows media files to be grouped in directories by project ft afso 
means that ciientappUcatton component 20 can use whatever file organization scheme 

it chooses. 

Each project object has a corresponding folder in the cache directory. Like Data 
Locatore, the directories are named with the ID of the, projedthey correspond to. Data 
Locator objects are stored within the folder of the project that contains them. 

Because media files can take up quite a lot of disk space, it is important that 
unused m ge< cleared. This is particularly tree when a hfcher quality file supercedes 
the current rendering «e. For example, a use, may workfora while with the preview 
version of an Asset, .hen later choose to downed the source rendering. At mis point 

. . a r»irt— ABset provides a method for clearing this 
the preview rendering is redundant. CRkt-ABBec pruv 

redundant data: 

// Cleat up the media we are no lon,ec using. 
// tieoi • ■* ' eieteLocalMedia 

pAsset > » e R Y s \\ 1 % REVIEH _ RE KD_CIASS, , 

ASSET COHP*ESSED_REHD_STATE ) ; 
B Asset->DeleteLocalHedia 
(ASSET PREVIEM_REHO_CIASS. . 
ASSET_DECOMPRESSED_REND_STATE I; 

«i« fr«m thP cache and deletes the file from disk or 
This call both clears the rendenng file from the cacne 

RAM. 

,t will be apparent to those skilled in the art that various modifications and 
variations can be made in the methods and systems consistent wHh the present 
invention without departing from the spirit or scope of the invention. For example, * all 
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of the constants in the invention described above were multiplied by the same constant, 
the result would be a scaled version of the present invention and would be functionally 
equivalent. The true scope of the claims is defined by the following claims. 
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vmmat .ft CLAIMED IS : 

1. Apparatus for sharing sequence data between a local sequencer station and at 
,east one remote sequencer station over a network via a server, the sequence 
data representing audiovisual occurrences each having descriptive 
characteristics and time characteristics, the apparatus comprising: 
' ' afi^t interface module receiving commands from a local sequencer statibn; 
a data packaging module coupled to the first interface module, the data 
packaging module responding to the received commands by 
encapsulating sequence data from the local sequencer station into 
broadcast data units retaining the descriptive characteristics and time 
relationships of the sequence data, the data packaging module also 
7 extracting sequence data from broadcast data units received from the 
server for access by the local sequencer terminal; 



a broadcast handler coup.ed.to the Mimm.^^-"***^ 
module, the broadcast handler processing commands received via the 

first interface module; 
a server communications module responding to commends processed b, the 
broadcast handler by transmitting broadcast data units to the sewer for 
distribution to at least one remote sequencer station, the server 
communications module also receiving data available messages and 
broadcast data units from the server; and 
a notification gueue handler coupled to the sewer communications module and 
responsive to receipt of data available messages and broadcast data unite 
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from the server to transmit notifications to the first interface for access by 
the local sequencer terminal. 

2. Apparatus as recited in claim 1 wherein the data packaging module 
encapsulates the sequence data into broadcast data units including an 
arrangement data unit establishing a time reference, and at least one track data 
unit having a track time reference corresponding to the arrangement time 
reference, each track data unit having at least one associated event data unit 
representing an audiovisual occurrence at a specified time with respect to the 
associated track time reference. 

3. Apparatus as recited in claim 2, wherein the sequence data produced by the 
local sequencer station includes multimedia data source data units and wherein 
the data packaging module encapsulates the multimedia source data units into 
at least one type of asset rendering broadcast unit, each asset rendering 
broadcast unit type specifying a version of multimedia data source data 
exhibiting a different degree of data compression. 

4. Apparatus as recited in claim i, wherein the server communications module 
respondsto commands processed by the broadcast handler by transmitting 

asset rendering-broadcast-units-of-a-selectedasset-rendering-broadcast unit-type 

to the server for distribution to at least one remote sequencer station 
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Apparatus as rented in daim 3. wherein the sequence data units produced by 
the iocai sequencer station inciude dip data unite each repreaenting a specified 
porfion of ****** data source date unit and wherein the data packing 
m odu,aancapsu,atesthecnpda.aunUsin to broadcaa«dipda.aunita. _ 



6 Apparatus as recited in da*, 5. wherein the data packaging module 
ancapsuia.es sequence data unite into broadcast dip event data units each 
presenting a specified portion of a multimedia data source date unit beginmng 
ataapecfcdtirnewimrespedtoanassocteted^okUmereference. 

7 Apparatus as recited in claim 6. wherein: 

». date packaging module encapsulates sequence date units into scope even. 
. , aateuniteeachhav^gaacopaeventtmereferenceeatabnahedata 

SD ec«ic time with reaped .o an assodated trade time reference; 
each scope even, date unit induding at teas, one timeline event date unK. each 
tim e,ine even, date unit having a timeline even. fime reference established 
a, a spedfic time with reaped to the associated scope even, time 
reference and including a. leas, one even, date unit representing an 
audiovteua, occurrence a. a specified time with resped to the associated, 
timeline event time reference. 



8 Apparatus aa recited in Cairn 1 . comprising a connect control component 
responsive to commands recetved from the local sequencer ataUon to establish 
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access via the server to a predetermined subset of broadcast data units stored 
on the s rv r. 



9. Apparatus as recited in claim 8, wherein the connection control component 
receives registration data from the local sequencer station and establishes 
access to a predetermined subset of broadcast data units stored on the server in 
accordance with permission data stored on the server. 

1 0. Apparatus as recited in claim 1 . wherein the data packaging module: 

encapsulates sequence data into first and second types of broadcast data 
units; 

responds to receipt of a message indicating the availability at the server of 
the first type of broadcast data unit by causing the server 
communications module to initiate a download of the first type of 
broadcast data unit without requiring authorization from the client 
application component; and 

responds to receipt of a message indicating the availability at the server of 
the second type of broadcast data unit by causing the server, 
communications module to initiate a download of the second type 
of broadcast data unit only after receipt of a download command 
from the client application component 
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Apparatus as recited in claim 10, wherein the first type of broadcast data unit 
compris s a nonlmedia broadcast data unit and the second type of broadcast 
data unit comprises a media broadcast data unit. 

Z. Apparatus for sharing sequence data between a local sequencer station and at 
least one remote sequencer station over a network via a server, the sequence 
date representing audiovisual occurrences each having descriptive 
characteristics and time characteristics and including multimedia data source 
data units, the apparatus comprising: 

a first interface module receiving commands from a local sequencer station; 
a data packaging module coupled to the first interface module, the data 

packaging module responding to the received commands by. . 

encapsulating sequence data from the local sequencer station into 

broadcast data units retaining the descriptive. character^ 

relationships of me sequence data. the datapa^ 

encapsulating the multimedia source data units into at least one type of 
asset rendering broadcast unit, each rendering broadcast unit type 

specifying a version of multimedia data source data exhibiting a different 
degree of data compression, the data packaging module also extracting 
sequence data from broadcast data units received from the server; 
a broadcast handler coupled to the first interface module and the data packaging 
module, the broadcast handler processing commands received via the 
first interface module; and 
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20 a server communications module responding to commands processed by the 

broadcast handler by transmitting broadcast data units to the server for 
distribution to at least one remote sequencer station, the server 
communications module also receiving broadcast data units via the server 
from the at least one remote sequencer station. 

13. Apparatus for sharing sequence data between a local sequencer station and at 
least one remote sequencer station over a network via a server, the sequence 
data representing audiovisual occurrences each having descriptive 
characteristics and time characteristics, the apparatus comprising: 

5 

a first interface module receiving commands from a local sequencer station; 

a data packaging module coupled to the first interface module, the data 

packaging module responding to the received commands by 

o encapsulating sequence data from the local sequencer station into 

broadcast data units retaining the descriptive characteristics and time 

relationships of the sequence data, the broadcast data units including 

custom broadcast data units, standard broadcast data units expressing 

the hierarchy of sequence data, and specialized broadcast data units 

_ 15 . , inc | U ding all attributes-of-standard^^ 

attributes, the data packaging module also extracting sequence data from 

broadcast data units received from the server; 
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a broadcast handler coupled to the first interface module and the data packaging 
module, the broadcast handler processing commands received via the 
20 first interface module; and 

. a server communications module responding to commands processed^yihe 
-~ " ■ broadcast handlefby transmitting broadcast data units to the server for 
distribution to at least one remote sequencer station, the server 
• communications module also receiving broadcast data units via the 
server from the at least one remote sequencer station and passing the 
received broadcast data units to the data packaging module. 



25 



10 



14. 



A method for sharing sequence data between a local sequencer station and at 
.east one remote sequencer station over a network via a server, the sequence 
data representing audiovisual occurrences each having descriptive 
characteristics and time dharactertef.es. the method comprising: 
receiving commands via a client application component from a user at a local 
sequencer station; 

responding to the received commands by encapsulating sequence data from the 
local sequencer station into broadcast data units retaining the descriptive 
characteristics and time relationships of the sequence data and 
transmitting broadcast data units to the server for distribution to at least 

one remote sequencer station; 
receiving data available messages from the server. 
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responding to receipt of data available messages from the server to transmit 

notifications to the client application component; 
responding to commands received from the client application component to 

request download of broadcast data units from the server; and 
receiving broadcast data units from the server and extracting sequence data 

from the received broadcast data units for access by the client application 

component. 



58 



WO 01/22398 



PCTAJS00/25977 



1/11 



Server 



^r-12 



16 

1 




Remote 
Sequence 
Station 



16 



Remote 
Sequencer 
Station 



14 

i 




Services 
Component 



•24 



20 

1 



22 

1 



Client 
Application 
Component 



Control 
Component 



USER 



in 



FIG. 1 



WO 01/22398 



FCTAJS00/2S977 



2/11 




FIG. 2 



WO 01/22398 PCIYUSOO/25977 



4/11 















E 








Oi 




Arran 





CO • 

o 



o 

CO 



o 



WO 01/22398 



PCT/US00/25977 



5/11 




■5- 



C 

f 
ui 



in 
O 



WO 01/22398 



PCT/USOO/25977 



6/11 





CO 



WO 01/22398 



7/11 



PCT/US00/25977 





WO 01/22398 



PCTAJSOO/25977 



8/11 















8, 































a. 



CO 

8 

c 

e 
e 



00 

O 

m 



c 

ur 



0) 

> 
ai 

b 



WO 01/22398 PCT7US00/25977 



10/11 



o 



d 



to 

CO 



1 

'S 

o 

I 
o 



O 
ti. 



"WO01/22398 



PCTAJS00/2S977 



11/11 



Rocket Object 




' r~Mix Tr ack < 

L- ■ ! — I 



FIG. 11 



INTERNATIONAL SEARCH REPORT 



Interna application No 

RCT/U6 00/25977 



A. CLASSIFICATION OP SUBJECT MATTER 

IPC 7 GlOHl/00 



According to Intemallonal Patenl Ctessfflcalton (IPC) or lo both national classification and IPC 



B. FIELDS SEARCHED 



Minimum documeniallon Bearched (classification system followed by classification symbols) 

IPC 7 G10H 



Documentation searched other than minimum documemaiton to Ihe extent mm wen teuments are Included In the Raids 



Electronic data base consulted during the Intemallonal search (name of data base and. where practical, search terms used) 

EPO-Internal , WPI Data 



C. DOCUMENTS CONSIDERED TO BE RELEVANT 



Category • Citation of document, with Indication, where appropriate, of the relevant passages 



Relevant to claim No. 



WO 94 11858 A (MULTIMEDIA SYSTEMS CORP) 

26 May 1994 (1994-05-26) 

page 3, line 22 -page 4, line 16 

page 11, line 3 - line 17 

page 18, line 2 -page 20, line 20; figure 

4 

US 5 952 599 A (DOUGHERTY TOM ET AL) 
14 September 1999 (1999-09-14) 
column 1, line 43 -column 2, line 3 
column 3, line 39 - line 55; figure 2 

EP 0 933 906 A (YAMAHA CORP) 
4 August 1999 ( 1999-08-04) 
paragraphs 

' 0006 ! , * 0007 ! , ' 0012 ! , ' 0013 ! , * 0021 ! - « 0032 ! , 
'0061!, .'0062!; figures 1,2 



1-14 



1-14 



1-11,14 



□ 



Further documents are listed In the continuation of box C. 



Patenl family members are listed In annex. 



* Special categories of cited documents : 

'A' document defining the genera) siate of the art which Is not 

considered to be of particular relevance 
•E* earlier document but published on or alter the International 

filing dale 

^•-document which may throw doubts on prtortty-Clalm(s) or 
which Is cited to establish the publication date of another 
Citation or other special reason (as specified) 

•O" document referring to an oral disclosure, use, exhibition or 
other means 

•P* document published prior to Ihe International fifing date but 
later than the priority date claimed 



•T* later document published after the International tiling date 
or priority date and not hi conflict with the application but 
cited to understand the principle or theory underlying the 
invention 

■X" cement of particular relevance; the claimed Invention 
cannot be considered novel or cannot be considered to 
-irwota:an Invents step when^ taken atone 



■V document o! particular relevance; Ihe claimed Invention 
cannot be considered to Involve an Inventive step when the 
document is combined with one or more other such Docu- 
ments, such combination being obvious to a person skilled 
In the art 

documenl member ot the same patent lamfly 



Date of the actual completion of the international search 

29 December 2000 



Date ot mailing of the intemallonal search report 

05/01/2001 . 



d mailing address of the ISA 

European Patent Office, P.B. 5818 Patentlaan 2 
NL - 2280 HV Rljswfjk 
Tel (+31-70) 340-2040. Tx. 31 651 epo nt. 
Fax: <+3W0) 340-3016 



Authorized officer 



Pulluard, R 



Foitn PCT/ISA/2I0 (second sheet) pu* 1B92) 



INTERNATIONAL SEARCH REPORT 





Fom>PCTflSA/ZiO(p8i<ml lomUy ««») (JuV '992) 



(12) INTERNATIONAL APPLICATIONTUBLISHED UNDER THE PATENT COOPERATION TREATY (PCT) 



CORRECT^) VERSION 



(19) World Intellectual Property Organization 
International Bureau 

(43) International Publication Date 
29 March 2W)^(29 ; 03^0M) 




(10) International Publication Number 

WO 01/22398 Al 



(51) 
(21) 



International Patent CUssification^: GlOHl/00 (72) Inventors: MOLLER, Matthew, D.; 4317 26th Stoeet, 

San Francisco, CA 94131 (US). LYUS, Graham; 2000 
Post #338, San Francisco, CA 94115 (US). FRANKE, 
Michael; 253 Mississippi Street San Francisco, CA 94107 
(US). 



International Application Number PCT/USOQ/25977 



(22) International 



(25) Filing Language: 



22 September 2000 (22.09.2000) 



(74) Agents: GARRETT, Arthur, S. et aL; Fronegan, Hender- 
son, Farabow, Garrett & Dunner, LLP, 1300 1 Street, N.W.. 
Washington, DC 20005-3315 (US). 



(26) Publication Language: 



(30) Priority Data: 

09/401318 



23 September 1999 (23.09.1999) US 



(71) Applicant: ROCKET NETWORK, INC. fUS/US]; 53 
Stillman Street, San Francisco, CA 94107 (US). 



English (gl) Designated States (national): AE, AG, AL. AM, AT, AU, 
AZ,BA,BB, BG,BR,BY,BZ»CA t CH,CN,CR,CU,CZ l 
DE, DK, DM, DZ, EE. ES. FL GB, OD, GE. GH, GM, HR, 
HU, ID, XL IN, IS, JP, KE, KG, KP, KR, KZ, LC, LK, LR, 
LS, LT, LU, LV, MA, MD, MG, MK, MN, MW, MX, MZ, 
NO, NZ, PL, PT, RO, RU, SD, SE, SG, SI, SK, SL, TJ, TM, 
TR,TT, TZ, UA, UG, UZ, VN, YU.ZA, ZW. 

[Continued on next page] 



(54) Title: SYSTEM AND METHOD FOR ENABLING MULTIMEDIA PRODUCTION COLLABORATION OVER A NET- 
WORK 




(57) Abstract: A system and method for collaborative mul- 
timedia prochictioD by users at different geographic locations, 
The usot produce sequencer data at a plurality of se^encex 
stations (14, 16) connected via a network (18). The sequencer 
stations (14, 16) encapsulate sequencer data units into broad- 
cast data units and upload and download broadcast data units 
to and from a server (12), in response to user co mmands re- 
ceived at the sequencer stations (14, 16). 



KILLS MW, MZ. SD. SVSZ. TZ. Ua. ZW). Eurasian 

patenuWAZ.BV.KG.KZ.MDWJ^Eur^ _ comctkm: 

Sa^tUBECH.Cy.D^ES.R.mOB.qR.^ (15) ^^No. 20Q001 of 17 May 2001. Section D 
IT. LU. MC, NU FT. SE). OAK patent (BE,B^..CG.. . seePCl 

amGA.GN.OW.MUmNE.SN.mTO). - ^^Wolto^^^/te TM* 

PiiBliilia: r " V T^'i-'Z/ " " ' - '"" 1 ning^eaehregidar lssue pftb' PCT Gazette. 
With international search report. 



PCT/US00/25977 

WO 01/22398 

-1- 

SYSTEM AND METHOD FOR ENABLING MULTIMEDIA PRODUCTION 
COLLABORATION OVER A NETWORK 
RAnKGROUNP OP THE INVENTION 

FiPlri of the Invention 

5 . The invention relates to data sharing and, more particularly, to sharing 

of multimedia data over a network. 

Computer technology is increasingly incorporated by musicians and 
multimedia production specialists to aide in the creative process. For 
example, musicians use computers configured as "sequencers" or "DAWs" 
10 (digital audio workstations) to record multimedia source material, such as 
digital audio, digital video, and Musical Instrument Digital Interface (MIDI) 
data. Sequences and DAWs then create sequence data to enable the user to 
select and edit various portions of the recorded data to produce a finished 
- product. 

15 Sequencer software is often used, when multiple artists collaborate in a 

project usually in the form of multitrack recordings of individual instruments 

gathered together in a recording studio. A production specialist then uses the 
sequencer software to edit the various tracks, both individually and in groups, 

to produce the final arrangement for the product. Often in a recording 
20 session, multiple "takes" of the same portion of music will be recorded, 

enabling the production specialist to select the best portions of various takes. 

Additional takes can be made during the session if necessary. 

Such collaboration is. of course, most convenient when all artists are 

present in the same location at the same time. However, this is often not 
25 possible. For example, an orchestra can be assembled at a recording studio 

in Los Angeles but the vocalist may be in New York or London and thus 

unable-toparticipate-inp^^ 

vocalist to participate from a remote studio linked to the main studio in Los 
Angeles by wide bandwidth, high fidelity communications channels. However, 

30 this is often prohibitively expensive, if not impossible. 

Various methods of overcoming this problem are known in the prior art. 
For example, the Res Rocket system of Rocket Networks, Inc. provides the 
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ability for geographically separated users to share MIDI ^ a ^°^^^^ m6t 
However, professional mu,«media production speaaHsts commonly use a 
. 1„ numLof wide* Known professional 
" ' Since they have extensive experience in using the interface of a particular 
5 ^arapacKage.Ureyareonenun-lingtoforegothehenef.sofsuch 

experience to adopt an unfamiliar sequencer. 

„ te therefore desirable to provide a system arid method for 
pro ,ess to ne, artists and multimedia product.cn specialists co»abora«a*on, 

10 sequencer software. 

c?i iMfc ^ARV OF TH F INVENTION 

Features and advantages of the invention will be set forth in the 

I. description which follows, and in par, wiii be apparent from the d*cnptton. or 

may be learned by praCce of the invention. The objectives and otter 

1S adltages o, the invention wii, be reaped and attained by . a system ^d 

, m e,hods parficulariy pointed out in the written description and claims hereof. 

., as well as the appended drawings. : 

"~ . ^accordance with the purpose of the invention as embodred and ; 

broadly described, .he invemion includes apparatus for sharing sequence 
20 dlleen a local sequencer station and a, leas, one remote sequencer 

H . inpl , uia a serv er the sequence data representing 
station over a network via a server, u ic ^ H 
.* u ovisue, occurences eech having descriptive characteristics and hme 
' aCerisKcs. The appara,us includes a ft* interface module = ng 
commands mom a local sequencer station and a data packaging module 
25 coupled tothenrs, interface module. The da« packing module respnds to 
M received commands by encapsulating sequence data from the local 
sequencers.a«on into broadcast data units retaining the 
characteristics and time relationships of the sequence data. The 
. pacing module also exacts sequence da« from broadcast da* un*s 

30 Lived from the server for access by the local sequencer 

apparatus further indudes a broadcas, handler coupled ,o ft. M ,n erf « 
mo du,e and me data packing module. The broadcas, handler processes 
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commands received via the first interface module. The apparatus also 
includes a server communications module responding to commands 
processed by the broadcast handler by transmitting broadcast data units to 
the server for distribution to at least one remote sequencer station, the server 
communications module also receiving data available messages and 
broadcast data units from the server. The apparatus further includes a 
notification queue handler coupled to the server communications module and 
responsive to receipt of data available messages and broadcast data units 
from the server to transmit notifications to the first interface for access by the 
local sequencer terminal. 

In another aspect the invention provides a method for sharing 
sequence data between a local sequencer station and at least one remote 
sequencer station over a network via a server, the sequence data 
representing audiovisual occurrences each having descriptive characteristics 
and time characteristics. The method includes receiving commands via a 
client application component from a user at a local sequencer station; 
responding to the'received commands by encapsulating sequence data from 
the local sequencer station into broadcast data units retaining the descriptive 
characteristics and time relationships of the sequence data and transmitting" 
broadcast data units to the server for distribution to at least one remote 
sequencer station; receiving data available messages from the server, 
responding to receipt of data available messages from the server to transmit 
notifications to the client application component; responding to commands 
received from the client application component to request download of 
broadcast data units from the server; and receiving broadcast data units from 
the serverand-extractlng se^ujBnce_datafrorn the received broadcast data 

--units-for-aGcess-by4hexlient-applicatioj5j:omponent 

It is to be understood that both the foregoing general description and 
the following detailed description are exemplarily and explanatory and are 
intended to provide further explanation of the invention as claimed. 

The accompanying drawings are included to provide a further 
understanding of the invention and are incorporated in and constitute a part of 
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m specification to illustrate embodiments ,f the invention and, together with 
the description, serve to explain the P**^^ 

mm i rinrnrnnH ^ CT » pnRAWINGS 

V " The accompanying drawings which P^*^ 1 
5 ^partof this specification illustrate embodiments of the ,nvent,on and .togethe 
5 part euros p advantages and principles of 

with the description serve to explain tne oojeui* 

, _ the invention. 

In the drawings: . 
" Fig . , is a block diagram showing system oonsistentwUh a*referred 

10 embodiment of the present invention; „ mmn o„ t 
Fig. 2 is a block diagram showing modules of the servrces component 

0f R9 ' Rg. 3 is a diagram snowing the hierarchicai relabonship of broadcast 



15 



20 



25 



30 



data units of the system of Fig. 1 ; 

Fig. 4 ,s a diagram showing the relationship between Arrangement 
nhiects and Track objects of the system of Fig. .1; 

* Floo is a diagram showing the rel ationship bafween T ract objecte and 
Event objects of the system of Fig. 1; : - shff 

Fig ; 6 is a diagram showing^re«a» 
Rendering objects of the system of Fig. 1; 

Fig 7 is a diagram showing the relationship between Cl,p o b) ects and 

Asset objects of the system of Fig. 1; 

F,g 8 is a diagram showing tne relationship between . Even. ob,ec*. 
dip Even, objects. Clip objects, and Asset objects o, .he system 

Rg 9 is a diagram showing the relationship between Even, objects, 
Scope Even. objeOs. and Timeline objecte of the system* Frg. 1 ; ; _ 

Fig. 10 is a diagram showing the relationship of Project objects and 
Custom objects of the system of Fig. 1; and 

Fig 1 1 is a diagram showing .he reladonshlp between Rocket objects, 
and Custom and Extendable objects of the system of Rg. It . 
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Computer applications for musicians and multimedia production 
specialists (typically sequencers and DAWs) are built to allow users to record 
and edit multimedia data to create a multimedia project. Such applications 
are inherently single-purpose, single-user applications. The present invention 
enables geographically separated persons operating individual sequencers 
and DAWs to collaborate. 

The basic paradigm of the present invention is that of a "virtual studio." 
This, like a real-world studio, is a "place" for people to "meet" and work on 
multimedia projects together. However, the people that an individual user 
works with in this virtual studio can be anywhere in the world - connected by 

a computer network. 

Fig. 1 shows a system 10 consistent with the present invention. 
System 10 includes a server 12. a local sequencer station 14, and a plurality 
of remote sequencer stations 1 6, all interconnected Via a network 18. 
Network 18 may be the Internet or may be a proprietary network. 

Local and remote sequencer stations 14 and 16 are preferably 
personal computers, such as Apple PowerMacintoshes or Pentium-based 
personal computers running a version of the Windows operating system. 
Local and remote sequencer stations 14 and 16 include a client application 
component 20 preferably comprising a sequencer software package, or 
"sequencer." As noted above, sequencers create sequence data 
representing multimedia data which in turn represents audiovisual 
occurrences each having descriptive characteristics and time characteristics. 
Sequencers further enable a user to manipulate and edit the sequence data to 
generate multimedia products. Examples of appropriate sequencers include 
Looic Audio from Emagic Inc. of Grass Valley. California; Cubase from 
stPinhgrg Soft- und Hardware GmbH of Hamburg, Germany; an d Prolgote 
from Digidesign, Inc. of Palo Alto, CA. 

Local sequencer station 14 and remote sequencer stations 16 may be. 
but are not required to be. identical, and typically include display hardware 
such as a CRT and sound card (not shown) to provide audio and video 
output. 
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Lccal sequencer station 14 also includes e connection control 
.JSSL* allows a user a, loca, sequencer s,a«on ,< to * to 

#4 eteh« channel assignment, and other similar 

imormauoM. T h* user name is the name of the 

also displays a new item: user name. The user nam 

aiso uia^'oy creatinq it on his local 

15 sequencer station. Tnrsnove Tracks are prefera bly 

connection thihasVtwo windows: incoming. Ghat 

,n response to parts ^ (o appear 

appear on the local - ^ P sffeen , music mmes trough 
25 in it. V the local user cl.cKs play on P ^ ^ 

has been working on his tracKs. oine. 
^^.user^e-^ 
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clicks a "Post" button of client application component 20 on his display screen, 
and all remote users in the virtual studio can hear what the local user is 
hearing -live. 

As shown in Fig. 1, local sequencer station 14 also includes a services 
5 component 24 which provides services to enable local sequencer station 14 to 
share sequence data with remote sequencer stations 16 over network 18 via 
server 12, including server communications and local data management. This 
sharing is accomplished by encapsulating units of sequence data into 
broadcast data units for transmission to server 12. 
10 Although server 12 is shown and discussed herein as .a single server, 

those skilled in the art will recognize that the server functions described may 
be performed by one or more individual servers. For example, it may be 
desirable in certain applications to provide one server responsible for 
management of broadcast data units and a separate server responsible for 
15 other server functions, such as permissions management and chat 
administration. 

Fig. 2 shows the subsystems of services component 24, including first 
interface module 26, a data packaging module 28, a broadcast handler 30, a 
server communications module 32, and a notification queue handler 34. 
20 Services component 24 also includes a rendering module 36 and a caching 
module 38. Of these subsystems, only first interface module 26 is accessible 
to software of client application component 20. First interface module 26 
receives commands from client application component 20 of local sequencer 
station 14 and passes them to broadcast handler 30 and to data packaging 
25 module 28. Data packaging module 28 responds to the received commands 
by encapsulating.sequence.daia..frornJgcal sequencer station 14 into 
-broadeast-data-unitsretain^ 



relationships of the sequence data. Data packaging module 28 also extracts 
sequence data from broadcast data units received from server 12 for access 
30 by client application component 20. 

Server communications module 32 responds to commands processed 
by the broadcast handler by transmitting broadcast data units to server 12 for 
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disttfbution to at least one remote sequencer station 16. Server 
communications module 32 aiso receives data available messages from 
ZTl2 and broadcast data units via server 12 from one or more remote 
Z near sfctions 16 and passes me received droadcas. data units 10 data 
6 5 ng moduie 26. In parliouia, senrer communications module recerve, 

lavaL, gesfTomserverlZlhatabroadcas. eteunftjfron, 

remote sequencer stottons 16, is avallabia a. the sewer, if .he available 

communions module requests mat .ha broadcast data unit be downloaded 
„ frTslr12. ,Hba avaiiabie broadces. da, uni. is o, a media ^pe se^ . . 
communions moduie request ma. .he broadcast data un« be downed 
from sewer 12 only after receipt of a download command from Cent 

applicatton component 20. ou l0 se „ er communica , io ns 

Notification queue handler is coupieu 

V ™H C .nr M elDt of data available messages from server 12 
15 module 32 and responds to receipt otuoio 

is muuu. .„ ,wi interface module 26 for access by client 

by transmitting notifications to first interlace mooui 

applicationcomponent20oflocalsequencerterm,nal14 .„„.„.,.. 
Typically, a user at, for example, ioca, sequencer stotton 14 w„, begma . 

' proieCo^reJrding muittmedia dato. This may be aocompiishe, through use 
20 of a microphone and video camera to record audio and/or visual 

« in the form o, source digtta, audio data ~e aud,o 
data stored on mass memory o, .oca, sequencer stabon 14. 

sequencer station 14 and storing the performance in toe form of MIDI date. 
25 Other types of multimedia data may be recorded. 

Once the data Is recorded, it can be represented in an "arrangement 
window on the display screen of local sequencer stabon 14 by client 
application component 20, typically a sequencer program. In a we,, known 
Iter, the user can sbleet and combine mulUpie recorded tracks either ,n 
30 Z en ireiy or in portions, to generate an anangement. Client 

omponent 20 thus represents this arrangement in toe form o, sequence data 
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which retains the time characteristics and descriptive characteristics of the 

recorded source data. 

When the user desires to collaborate with other users at remote 
sequencer stations 1 6, he accesses connection control component 22. The 
user provides commands to connection control component 22 to execute a 
log*ih procedure in which connection control component 22 establishes a 
connection via services component 24 through the Internet 18 to server 12. 
Using well known techniques of log-in registration via passwords, the user can 
either log in to an existing virtual studio on server 12 or establish a new virtual 
studio. Virtual studios on server 12 contain broadcast data units generated by 
sequencer stations in the form of projects containing arrangements, as set 

forth in detail below. 

A method consistent with the present invention will now be described. 
The method provides sharing of sequence data between local sequencer 
station 14 and at least one remote sequencer station 16 over network 18 via 
server 12. As noted above, the sequence data represents audiovisual 
occurrences each having a descriptive characteristics and time 

characteristics, 

When the user desires to contribute sequence data generated on his 
sequence station to either a new or existing virtual studio, the user activates a 
POST button on his screen which causes client application component 20 to 
send commands to service component 24. A method consistent with the 
present invention includes receiving commands at services component 24 via 
client application component 20 from a user at local sequencer station 14. 
Broadcast handler 30 of service component 24 responds to the received 
commands by encapsulating sequence data from local sequencer station 14 
--■into-broadcast-data-units-retaining-the-descri^ 
relationships of the sequence data. Broadcast handier 30 processes received 
commands by transmitting broadcast data units to server 12 via server 
communications module 32 for distribution to remote sequencer stations 16. 
Server communication module 32 receives data available messages from 
server 12 and transmits notifications to the client application component 20. 
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Server communicaUon module 32 responds .0 commands received .mm client 
'XL component 20 .o repues. download of broadcast da* .n* 

1 Z*» se.er.rom «- * « re ™ te Stat '° n - ^ , 

• Lived from server 12 for access by den. application component 20 

■ manipu,ating sets of broadcast data managed and persisted by server * * 

I ^n— men t, ser^ces compon en, 24 uses an objected 
10 data mode, managed and manipulated by data pacing modu 28 to _ . 

' objects created by services component 24 from sequence 
feline a hierarchy and map interdependent of sequence data tn toe 

„ ProieCt Fig. 3 shows the high ,eve, conminment hierarchy for object " 
muting broadcast data un*s in ,e preferred emboolrnent Each 

aWbutes and perform operations on the object. Cop.es of all broadcast 

objects are held by senrices component 24. 
,„ " ' Broadcast objects are created in one of wo ways: 

. . Creadng object locally and broadcasting them to server ■ 

Client applica^on component 20 creates broadcast objects tlocalyby ca,ng 
■ create metoods (set tonh in detail in the Appendix) on other obiects ,n toe 

^ " ttces component 24 uses a notion system of notficadon gueue 

Notificatons allow services component 24 to tell the 
changes in the states of broadcast objects. 
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Client application 20 is often in a state in which the data it is using 
should not be changed. For example, if a sequencer application is in the 
middle of playing back a sequence of data from a file, it may be important that 
it finish playback before. the data is changed. In order to ensure that this does 
5 not happen, notification queue handler 34 of services component 24 only 
sends notifications in response to a request by client application component 
20, allowing client application component 20 to handle the notification when it 
is safe or convenient to do so. 

At the top of the broadcast object model of data packaging module 28 
1 0 is Project, Fig. 3. A Project object is the root of the broadcast object model 
and provides the primary context for collaboration, containing all objects that 
must be globally accessed from within the project. The Project object can be 
thought of as containing sets or "pools" of objects that act as compositional 
elements within the project object. The Arrangement object is the highest 
5 level compositional element in the Object Model. 

As shown in Fig. 4, an Arrangement object is a collection of Track 
objects. This grouping of track objects serves two purposes: 

1, It allows the Arrangement to define the compositional context of 

the tracks. 

20 2 . It allows the Arrangement to set the time context for these 

tracks. 

Track objects, Fig. 5. are the highest level containers for Event objects, 
setting their time context. All Event objects in a Track object start at a time 
relative to the beginning of a track object. Track objects are also the most 
25 commonly used units of ownership in a collaborative setting. Data packaging 
module 28 thus encapsulates the-sequence data into broadcast data units, or 

^BjecfsTihWd^ 

least one track object having a track time reference corresponding to the 
arrangement time reference. Each Track object has at least one associated 
30 event object representing an audiovisual occurrence at a specified time with 
respect to the associated track time reference. 
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Th, sequence dale produced by client applied component 20 of 
.oca, eeo.ue.cer station 14 includes multimedia data source data un, a denved 
. ^"ordedda*. Typically mis recorded data - be 

audio-da*. or digital video data, though any type of data can be recorded and 
r ttd These -Lmedia data sourca data un* used in the Pro.ec> are 

Fio 6 sho« an Asset object has an associated set of Rendenng ob,eots 
SIS use these Rendering obieC* to represent different V.e^a 
p La pfcee of media, thus Asset end Rendering ob,e«s ^ra designated 

media broadcast data units. A„ broadcast data units other than Asset and 
Rendering objects are o, a type designated as non-media broadcast data 

UPHS ' Ea * Asset object has a spatial Rendenng object mat represents , he 
original source recording of the data. Because digital media data ,s often very 
Z this original source dale may>,ever be distributed across the neb**,. 
o^ad. compressed versions o, me data will be sen,. ™— ^ 
versions are represented as alternate Rendenng ob,ects o, the Asset object 

se «ng and manipuladng Ihese Rendehng ^^^^ 
me ans o, managing venous versions of source data, grouping £ «n 
common composUional element. Data packaging module 28 mus 
encapsulates the multimedia source objects into at least one type of asset 

version of multimedia data source data exhibiting a dmeren, degree of data 

M "hTslguence data units produced by client application co^pon-i 20 
of local sequencer station 14 include clip data units each repressing a 
' ' Led portion of e multimedia data source data unit. Data pacxag, g 

30 used 10 reference a section of an Asset object, as 

primary purpose of the Clip object* to define the porhons of the Aase ob ect 
ha. are compositional* relevant For example, an Asset objee. represent^ 
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a drum part could be twenty bars long. A Clip object could be used to 
reference four-bar sections of the original recording. These Clip objects could 
then be used as loops or to rearrange the drum part. 

Clip objects are incorporated into arrangement objects using Clip Event 
5 objects. As shown in Fig. 8, a Clip Event object is a type of event object that 
is used to reference a Clip object. That is, data packaging module 28 
encapsulates sequence data units into broadcast data units known as Clip 
Event objects each representing a.specified portion of a multimedia data 
source data unit beginning at a specified time with respect to an associated 

10 track time reference. 

At first glance, having two levels of indirection to Asset objects may 
. seem to be overly complicated. The need for it is simple, however, 
compositions are often built by reusing common elements. These elements 
typically relate to an Asset object, but do not use the entire recorded data of 
1 5 the Asset object. Thus, it is Clip objects that identify the portions of Asset 
objects that are actually of interest within the composition. 

Though there are many applications that could successfully operate 
using only Arrangement, Track, and Clip Event objects, many types of client 
application components also require that compositional elements .be nested. 
20 For example, a drum part could be arranged via a collection of tracks in 
which each track represents an individual drum (i.e., snare, bass drum, and 
cymbal). Though a composer may build up a drum part using these individual 
drum tracks, he thinks of the whole drum part as a single compositional 
element and will-after he is done editing-manipulate the complete drum 
25 arrangement as a single part. Many client application components create 
.folders for these tracks, a nested.part that can then be edited and arranged as 



— a-single unit — — : : • 

In order to allow this nesting, the broadcast object hierarchy of data 
packaging module 28 has a special kind of Event object called a Scope Event 

30 object, Fig. 9. 

A Scope Event object is a type of Event object that contains one or 
more Timeline objects. These Timeline objects in turn contain further events, 
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u «-, e m ^moe Event objects are thus very similar to 
providing a nesting mechanism. Scope tvem u j 
P 9 u tc . thp ScoDe Event object sets the start time (the tune 
Arrangement objects: the Scope tvem ouj 

r^ntpytHor all of the Timeline objects it contains. 

5 * « these T,me„ne objects contain are a„ relative to the start le oMhe ■ 
Scope Even, object Thus, rtata pacKagtng moduie 28 encepsu^s ^ 
- . , s «, ence data units into Scope Even, «. -» •**""»" 

Event time reference established at a specific time wnthtespeC to an 

«ra<* time reference- Each Scope Even, object inciudes a, leas, 

associated timeline event time reference. ^ 

A Project object contains zero or more Custom-Object, F, 9 . 0.^ 

20 '^ad^b^^^ 

28 contains,wospecia,objects: roCe, objec, and exten abt.Ait 

25 ; ^"bieobjecfsareobjectsfhatcenbeextendedbyolien, 

„„ m ,20 AS shown in Fig-1 1 ■ these objects constitute 

tridcT l un "Jhich express the hierarchy o, sequence data, 
standard broadcast ^ Rendenng 

30 objects. Tneexienu* broad cast data units for 
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packaging module 28 to implement a specialized object called a MixTrack 
object, which includes all attributes of a standard Track object and also 
includes additional attribut . s. Client application component 20 establishes 
the MixTrack object by extending the Track object via the Track class. 
5 As stated above, Extendable broadcast data units can be extended to 

support specialized data types. Many client application components 20 will, 
however, be using common data types to build compositions. Music 

sequwce^ 
and MIDI data types. 

1 o Connection control component 22 offers the user access to 

communication and navigation services within the virtual studio environment. 
Specifically, connection control component 22 responds to commands 
received from the user at local sequencer station 14 to establish access via 
12 server to a predetermined subset of broadcast data units stored on server 

15 12. Connection control component 22 contains these major modules: 

1. A log-in dialog. 

2. A pass-through.interface to an external web browser providing 
■ -•- access- • 

to the resource server 12. 
20 3. A floating chat interface. 

4. A private chat interface 

5. Audio compression codec preferences. 

6. An interface for client specific user preferences. 

- The log-in dialog permits the user to either create a new account at 
~2S server 1 2 or log-in to various virtual studios maintained on server"! 2 by 
entering a previously registered user name and password. Connection 
control component 22 connects the user to server 1 2 and establishes a web 
browser connection. 
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Once a connec.ion.is established, ft. user cen search through 
ava .«e vi rt ua, sn.dios on server 12. specify a studio to "enter; and 

exchange chat messages with Cher users 1— - seguance ^ons 16 
-through a chat window. 

,„ particular, connect conhol component* passes command. .0 
^co m ponenU4.««ch exchanges messages with server 12 viewer 

communion module 32. Prefe^ chai' maisSges are implement via a 

Multi User Domain. Object Oriented (MOO) protocol. 

fro m server 12 .or processing by ciien. applicadon component 20 and 
connection confroi componen. 22. This communicacon is in .he form o, 

from server 12 .o achieve e spednc-funCio, The .uncdons performed by 
15 se^er communicadon module 32 include downing a single object, 

loading an object and ,.s children, downioeding media dale,, uploading 

' loggl „g in «o server 12 to access dam, and locadng a s.ud,o. 

^r-hiovPd bv a plurality of message types, 
■ ' * These functions are achieveo oy a piu.a y 

20 described below. 
ack 

This Is a single acknowledgement of receipt. 

NACK 

This message is a no-acKnowledge and inciudes an error code, 
the object. 
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Request object and children 

Thlsmessage identifies the studio. Iden».f,e8 the p^JertcontaWng th*ob)ea Identities objed wlmse dtfd 

objects end self Is to be downloaded, and Identifies the class of object. 
Broadcast Start 

5 Thte^ssageldenUfies lhe studlo-and Identifies the project being broadcast. 

Broadcast Create 

This massage Identifies the studio, Identifies the project containing the obiect. identifies the object being 
created, and contains the object's data. 
Broadcast Update 

1 0 This message Identifies the studio, identifies the project containing the object identifies the object betog 

. . gpdated. Identifies the cless of object being updated, end contstns the object's data. 
Broadcast Delete 

This messege identifies the studio. Identifies the project containing the object identifies the object being 

deleted, and identifies the class of object being updated. 

15 Broadcast Finish 

This message identifies the studio, and identifies the project being broadcast 

Cancel transaction 

This message cancels the current transaction. 

Start object download 

20 This message identifies the object being downloaded in mis message, identifies the dess of object 

identifies the parent of the object and contains the object's data. 

Single object downloaded 

object data. 
25 Request media download 

Th* message identifies me studio, identifies me project containing the object, identifies me rendering 
object associated with the media to be down.oaded. and identifies the class of object (always Rendering). 
Broadcast Media 

This messege Identifies me studio. Identifies the project containing me object identifies the Medle object to 
30 be uploaded, identifies me dess of objed (always Media), identifies me Medie's Rendering peren, object 

and contains Media data. 
Media Download 

This message identir.es the rendering objed essodated with me media to be dowdoaded. Identifies the 
class of object (always Rendering), and contains the media data. 

35 Request Tlmestamp 

This message requests a timestamp. 

Response Timestamp 
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imnia! YYYYMMDDHHMMSSMMM (Year.-Month. Day of 
This message contains a emeslamp in the format YYYYMM 

Month. Hour. Minute. Second. Milliseconds). 

Request Login nttemDtinQ to Login Bnd provides an"MD5 digest lor security. 

This message Identifies the name of user attempting to Log 



Response SSS Login 

Is messag 



teR a Ml8tered . Pro - version; and provides a Session token, a URL for 
This message indicates if a user has a regjsterea 



the i 



Request Studio Location • - - - " • - " 

1Mb message IjNiWta* (]P '^^J^^'^"'** 8 ^^^ - ^^^^*^^" lf ~*^~^ — * ' 
1 o the community and studio names. 

Response Studio Location . • 

. h nnft for the MOO. and the address of the MOO. 
This message identifies the studo. the port for me Muu. 

15 downloaded, and identifies the dass of ObjecL 

identifies the parent of object 

20 trough a satot^ 

conned in a Cass librae In the preferred en*od,men, these Casses ere 
fomented in straighffoKvaru. cross-p,a«onm C~ and require no specie, 

A sequencer manufacturer integrates a ciien, applicaaon component 20 
25 to services component 24 oyiinKing the ciassliPrary to source code of ciien, 
app „ca,ion component 20 in a weii-Known manner, using for exampie, vi.ua, 
C++ for Windows ap P „ca t ,on or MetroworKs Codewarrier( P ro Reiease 4, for 



Macintosh application: 



s. 
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Exception handling is enabled by: 

Adding Initialization and Termination entry points to client 

application component 20 ( initialize and _terminate), 

- Adding "MSL RuntimePPC++.DLL" to ciient application 
5 component 20, and 

Add "MSL ARpRuntime,Lib" to client application component 20 
Once these paths are specified, headers of services component 
24 simply are included in source files as needed. 
A detailed description of the classes of the class library necessary to 
10 implement a system consistent with the present invention is set forth in the 
Appendix. 

To client application component 24, the most fundamental class in the 
first interface module 26 is crK.serv.ces. It provides methods for performing the 
following functions: 
15 . Initializing Services component 24. 

Shutting down Services component 24. 

Receiving Notifications from Services component 24. 

Creating Project objects. 

Handling the broadcast of objects to Server 12 through services 

20 * r ' component 24. 

" ~'~ ~ Querying for other broadcast object interfaces: 

Each implementation that uses services component 24 is. unique. 
Therefore the first step is to create a services component 24 class. To do 
this, a developer simply creates a new class derived from 
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ClaSSCHyRXtServices : p*U« CrKtServices 

I 

pubUC: cMyWctServicesl); 

„ virtual -CMyWctServices <> i 

"etc - 

); 



*„ .ppfcrfncW** to Se^es.componenmby^ng an 



10 try 

I 



« y Roc k etSer^^ .. 
!,MyRocmService B ->Initiali«el), 

15 latch "( CRiMctExceptiont e) 

L , . /./ initialise Failed 

20 ' autcmateally performs all operattons necessa^to 

• a « n n with services component 24-for client application 
initiate communication with serviw k 

component 20. -.. ■; 

„ 0 nt 20 disconnects from Services component 
Client application component 20 disconn 

25" . ' 2 4*y<leteting ^~^^" ?n8 ^5"-:; ;lL' ,; 'V " ' 

' . .„> 24 "Class was created, delete it 
//if a services component 24 Cias 

[i (m^RktServices I- MLU 
i*' .delete m_p!*tservices; . 

Services component ^ win 

w n rpnistered by the client application. cRktservices 
objects that have been registereo oy . ,,,-.>: 

provides an interface for doing this: , 
35 try 

( // Register for our. types , of. ^^SmOKtavan ), 

catch ( CrktExceptioni e) 
( // initialize Failed 



40 



45 } 
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Llke cHkcsorvic... all broadcast objects have corresponding crw interface 
implementation classes in first interface module 26. It is through these o*t 
interface classes that broadcast objects are created and manipulated. 
Broadcast objects are created in one of two ways: 
5 . Creating objects locally and broadcasting them to the Server. 

Receiving .a new objects from the server. 
There is a three-step process to creating objects locally: 

1 . Client application component creates broadcast objects by 
calling the corresponding create o methods on their container 

10 object. 

2. Client application component calls createMctmterf aee < > to get an 

interface to that object. 

3. Client application component calls aoetserviee.. , B ro«dea B to to 
. update the-server with the.se .new objects. 

15 Broadcast objects have creat.o methods for every type of object they 

contain. These create,, methods create the broadcast object in services 
component 24 and return the ID of the object. 

For example, a*«.rvice 8 has methods for creating a Project. The 
following code would create a Project using this method: 

20 !o"o=KeJp^nWcet=n tor possiHe error conditions 

----- try — • ; ; - 

// attempt to create project 
25 PP S^t5ervices : : Instance ( ) - >CreateWctProjectInterf ace 

CRktServices: j Instance { J ->CreateProject 0 ); 

30 // user created, set default name 

pProject->SetNaroe( -New Project- ) : 



35 



} // try 

catch ( CRktExceptionfc e ) 
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deiese pttojeet: 
e.aeporiRktErrorO ; 
re:;m false; 



10 



15 



return iai.H«=» , 

To create a Track, client application component 20 calls the 

has MA* -ate Us specific types o, child Proadcas, o bj ects.. 

„ ^necessary (nor desirableyte call --r^y— » 

the user hits a "Broadcast" button, forinstance). 

Because services component 24 Keeps tracK of and manages all 
changed broadcast objects, client action component 20 can taKe 

us ers to choose when ,o share their conthbutions and changes ^r 

users connected to the Project. — - - 

N0 ,e «ha, (unlifce ^^daU modaHmeHpce objec* arena. - 

parent object. 

Client app.icaf.on component 20 can get ^interface objects at an, 
, time . The ohjects are not de,eted fro. data pacing modu.e 2B until the 
Rem ove„ method has successfully completed. 

„ nont , n accesses a broadcast object as 
Client application component 20 accesses a 



25 
30 



follows: 

,/ O ot interf.ce to the new prelect 
// set name* 

{ . CR *cPtr < ^""^"^r^CrrateJOctProiectlx.ter.ac. (Preset, , 

MyProject->BetMwnet 



) n try , £ « ) 

catch ( CRXtExceptiont c ) 

{ 
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e . Bopor tlOstBrror ( ) i 

} The cRUPtro template class is used to declare auto-pointer objects. 
This is useful for declaring interface objects which are destroyed 
5 automatically, when the cmptr goes out of scope. 

To modify the attributes of a broadcast object, client application 
component20 calls the access methods defined for the attribute on the 
corresponding CRkt interface class: 

// Change the name of my project 

10 ^g^^^^ an associ a t ed Editor that is the only user 
allowed to make modifications to that object When an object is created, the 
user that creates the object will become the Editor by default. 

Before services component 24 modifies an object it checks to make 

1 5 sure that the current user is the Editor for the object If the user does not 

have permission to modify the object or the object is currently being broadcast 

to the server, the operation will fait 

Once created, client application component 20 is responsible for 

deleting the interface object: 

20 deletepTrack; 

Deleting a*t interface classes should not be confused with removing 
the object from the data model. To remove an object from the data model, 
you call the object's Eeaoveo method is called: 

P Track-»Reniove(): // remove from the datB model 

25 Interface objects are "reference-counted." Although calling *«ovet, will 

effectively remove the object from the data model, it will not de-allocate the 
interface to it. The code for properly removing an object from the data model 



is: 
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CRXtTracK* pTracfc; 

//.Create interface ... /; rBmove from the data model 

P Track->Re«ove(l , inlorface object 

delete pTraokj " OB, ° 

5 " or using the cmxtpcr Template: 

CRkt^tr < CRrktTracX > pTracX, - 
// Create Interface - 

^^...•.'.W—t.. method is caned. ■: 
15 umeuserdoesnothavepe—temodifytheoblectora 

bro adcas, is in progress, .ha operation « fell, throwing an exception. 

- ^adcastohiac.arano. san. and— d to San,ar12 U nt,i tha _ 

interface method is caUad. This aiiows users*™* 

cha nges iocaiiy betore commit than, «a ft. server and o«har users. The 

app^on component 20 to proceed even as data is being upioaded. 
1: "to ansura that its database remains consistent during the broadcast 
■ p.ocadure.sawicescontponent^doesncaiiawenvobiectstobemodi.ied 

wM eabroadcastisinprogress. Whan M changed obiacts have bean sent to 

t „ notification will be sent to the cl.ent 
25 the server, an onBroadcwtcompi.te noui.oo 

application. 

„ object mode, before committing them to server 12 by caiiing 

♦;«n i C railed the objects revert back 
When this operates called. 

n hPfnre the last broadcast. (This operation does not 
30 to the state they were in before the lasxu 

apply to media data.) 
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Roiibaeko is a synchronous method. 

Client application component 20 can cancel an in-progress broadcast 
by calling crxtse^coe, .canceXBrcdc-tn . This process reverts all objects to the 
state they are in on the broadcasting machine. This includes all objects that 

5 were broadcast before caneeiBroadcaato was called. 
CMOMtoitfl is a synchronous method. 

Notifications are the primary mechanism that services component 24 
uses to communicate with client application component 20. When a 
broadcast data unit is broadcast to server 12, it is added to the Project 

10 Database on server 1 2 and a data available message is rebroadcast to all 

other sequencer stations connected to the project. Services component 24 of 
the other sequencer stations generate a notification for their associated client 
application component 20. For non-media broadcast data units, the other 
sequencer stations also immediate^ 

15 broadcast data units; for media broadcast data units, a command from the 

associated client application component 20 must be received before a request 
for download of the available broadcast data units is generated. 

Upon receipt of a new broadcast data unit, services component 24 
generates a notification for client application component 20. For example, if 

20 an Asset object were receivedrthe o»cr..te*sae«:cn*i. t .o notification would be 
generated. 

All Notifications are handled by the cwctsorvic. instance and are 
implemented as virtual functions of the ooctservicea object. 
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Koiifcacco is a synchronous method. 

Cent application component 20 can cancel an in-progress broadcast 

' This includes all objects that 

state they are in on the broadcast^ machme. Th.s 

5 were broadcast before cnc-moa^o was called, 
c^^t ,. is a synchronous method. 

Notifications are the primary mechanism that senses component 24 
uses to communicate with client application component 20. When a 
br0 adcast data unit is broadcast to server 1 2, it is added to the Project 

orv,pr 1 2 and a data available message is rebroadcast to all 
10 Database on server 1 2 ana a uuw 

artpri to th e nroiect. Services component 24 of 
other sequencer stations connected to the proje 

„ Q nArate a notification for their associated client 
the other sequencer stat»ons generate a notmc 

nont20 For non-media broadcast data units, the other 

- application component 20. r-ornun 

- - veneer stations aiso imm.edia.ety revues, dorado, me avaiiabte 

16 ' broad cast data units; .or media broadcast data unite, a command from *e 
» abated cUen, application component 20 must be received before a reo.ues, 
f0 r download of me available broadcast data units is flenerated. 

■ Upon receipt of a new broadcast data unit, sauces component 24 

g enera.es a notion for Cent application component 20. For example, if 

n ™i-«o notification would be 
20 an Asset object were received, the toft .. t — » 

generated. 

: implemented as virtual functions of the ^se^c object. 
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To handle a Notification, client application component 20 overrides the 
corresponding virtual function in its a*ts.rvic.e class. For example: 
class CMyRktServices : public CRktServices 
5 < 

// Overriding to handle OnCreateA8setGori!Pleta ( NoUnc8ttons 
virtual void onCreetteAeeatCowpiete ( 
eonat WctobjectldTypes robjectld, 
conat RktObjectldTypet rParentObjectld ; 

When-client-applicatioR iCoropoiMiit.2X)T!^§to J5^«pns.>ria 
notification queue handler 28, these overridden methods will be called: 

RfctNeetType 
15 CMyRktSorvicBBnOnCxcatoASDetStart [ 

const RktObjectldTypet 



10 



^onatVobi.ctldTyp... rPax.ntObJ.etId , 
{ 



20 ( try 



// Add this Arrangement to My Project 
i£ * ^^^jTJStLt < rParentOMectld-rObj.ctld,, > ,i try 
25 catch ( CRktException* e ) 

e.ReportRktBrrorO ; 
return ROCKET QUEUE_DO_NKST/ 

30 } 

.. .... . 

not be changed. For example, if client application component 20 is in the 
middle of playing back a sequence of data from a file, it may be important that 
35 it finish playback before the data is changed. 

In order to ensure data integrity, all notification transmissions are 
requested client application component 20, allowing it to handle the 
- -■ -notification^-^ is available, a_ 

message is sent to client application component 20. 
40 On sequencer stations using Windows, this notification comes in the 

form of a Window Message. In order.to receive the notification, the callback 
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window and notion message must be set This is done using** 
aa tSe r vi=es ! :Se U .ataHot ifi «tio D H M .a 1 er(,ma»>od: 

•^SS; receive me ^sg.koii^oh.pekdxkg 

10 message whenever mere are nations present on .he even, queue o, 

queue handler module 24. 

Client application component 20 would then oall 

^.Services: :Pro=eas N ext B aCa S o t i=etionO .0 instruct services _ 
component 24 to send notions for the next pending data nooteaUon: 

tteouBBt Notification. 

15 i^ZZSZZ^^ ... . 

I ~ ,. ^BT-nceBBHextDataMofcifieationO > 

1 CMyRktServicea.-.UxfltauceO^ProcBBHex 

> _ ^^tificationO causes services component .24 to 
20 PrpcessNextDataNotiEicatiouv/ 

re move the no«ica*on .rem the queue and call the corresponding notation 

handler, which Cent application component 20 has overridden^ its 

implementation of cRktservicee. 

On a Macintosh sequencer staaon. client applicaaon component 20 

25 places a Call tO CrktSexvieeai : 

ind then overrida the CHXtServices. . 
notification* <> in th.ir ldl a , 
., ^ataKotmc.tionAvailaWeO ^"^^ che event notification 

// This method called when data avai _ _ . 

voi^tS.rvicea.^a^otieic.tion.v.aahle,, 

{ 



30 

{ 

try 

{ proeesaNaxtDataMotlficatlonOt 
35 Ltch < OOctLoglcException a ) 

* e.ReporcRktErrorOi 
} 

40 } 
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As described in the Windows section above, PeocoMNoxeoacHotieieationo 
instructs services component 24 to remove the notification from the queue 
and call the corresponding notification handler which client application 
component 20 has overridden in its implementation of cwctservie*.. 

5 Because notifications are .handled only when client application 

component 20. requests them, J3^^|on hw^_ ° f services 
component 24 uses a "smart queue" system to process pending notifications. 
The purpose of this is two-fold: 

1 . To remove redundant messages. 

10 2 . To ensure that when an object is deleted, all child object 

messages are removed from the queue. 

This process helps ensure data integrity in the event that notifications 
come in before client application component 20 has processed all notifications 

-on-the queue 

1 5 The system of Fig.1 provides the capability to select whether or not to 

send notifications for objects contained within other objects. If a value of 
RocEBT.gustB.i.o.HEST is returned from a start notification then ail notifications for 
objects contained by the object will be sent. If «w«T_wi««.«»jwr_«« is 
returned, then no notifications will be sent for contained objects. The 

20 cr..«<T»c»pi.t. notification will indicate thatthe object and all child objects 

have been created. 

For example if client application component 20 wanted to be sure to 
never receive notifications for any Events contained by Tracks, it would 
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. n method and have it return 
override the o»cr«aceproj«ctst.rt<> meinoa anu 

aocKET_orens_«»_KOT_H88T: 

5 v! . , ...^ , ,^ ,. ,,,/ 

// don't aendia notif toattons *W 

// arching contain.* by tbia •»«B?«J--" --- 

10 Saturn ROCKST-OOOTEiDO^MBSX, ,.. 

) : - n -notification parse the objects conta.ned by the 

And in the craataTracaCaBplaeoO,, UQimcauvn y . .. 

' : track: 

15 S^LtSorviceo. .OndraateProjactC 

omplete ( 

conBt WctObjeetldTypB* 

objectld. 

const RktQbjectXdTypefc 

20 -—J ; referred emb0 di m ent. predefined broadcast objects are used 
whoever poss.ble. By doing mis, a common interchange standard is - 
supported. Most dient application component 20 will be able to make _ 

25 are times, howevo, when a ciient appiicafion component 20 wii, have to taitpc 

objects to its own use. 

The described system provides two primary methods for creating 
custom and extended objects, if client application component 20 has an 
object which is a vadation of one o, the objects in the broadcast object mode,. 
30 it can choose «o extend the:broadcas, object. This permits retenfion of al, of 
me atmbutes, methods and conftmen. of the broadcast object, wh,,e 
taiiodng i, to a specific use. For exampie. if client appiicafion component 20 
hasa type of Track which holds Mix information, it can extend the Track 
Ohjecto hokt atuibu.es which apply to the Mix Track impiementafion. All pre 
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defined broadcast object data types in the present invention (audio. MIDI, 
MID! Drum, Tempo) are implemented using this extension mechanism. 

The first step in extending a broadcast object is to define a globally 
unique RktExtendedDataldType: 

5 //a globally unique ID to identify my- extended. data ; jyPJ-_ 

const RktExtendedDataldType CIluiUu Lid r tff_^TB»DgD^TIUiCK^TTR^IO 
( -14A51841-B61B-lld2.BD7B.b060979C492B*' ) t 

This ID is used to mark the data type of the object. It Mov^seimces 
component 20 to know what type of data broadcast object contains. The next 
1 0 step is to create an attribute structure to hold the extended attribute data for 
the object: 

etruct CKyTrackAttributee 

^ CMyTracfcAttributes () i 

15 Int32Type m aMyQuantisej // tny extended data 

}t 

// Simple way to initialize defaults for your attributee ie 
// to use the. constructor for the struct 
20 CMyTrackAt tributes 1 1 CMyTr acXAt tributes ( J 

^ n nMyOuantixe - kMyDef aultQuantisej 

) 

foinit^ 

25 the data type Id, the data size, and the data: 

// sec my attributes 

CKyTrackAttributes myTracxAttributes / 
jnyTracxAttributoB.m^nHyQuantise » 16/ 

30 try 

// Sec the extended data cype 

P Track->SetDataType( MY_BXn£NDED_TRACK_ATTR_ID ) / 

35 // Set the data (and length) 

lnt32Type nSise - siieof (myTracklwt tributes ) i 
Track- >SetData { fcnyTrackAt tributes , anSise ) i 

L 



~ catch { CRktExeeption e ) 
40 { 

e . Repor tRktError ( ) j 



1 When a notification is received for an object of the extended type, it is 
assumed to have been initialized. Client application component 20 simply 
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iu interface and use its values as 
requests the attribute structure from the em .nterface 

necessary. 

« m see if we understand it. 
// Check che data type, to see it 
RktBxtendadDataldType dataType ); 
5 ^TMC^>OetDataType ( 

/./...Get -the ..Data,.. Upon return. 

// size of the data. Ammm s t cMvTrackXttributea )j 

' custom ObjecK are used to create propria* objects *h,ch * « 
dlre c,iy map ,o objects in me broadcast obiectmode. o,-data. pacing 
module 28. A Custom Data Object is a broadcast.object which holds arbitrary 
bina ry data. Custom Data Objects atso have a« which spec* the type 

ofda .a contained by the obiec. so mat ^ , 

ent 54 does provide all of the normal services. 

15 object. Services component 24. does pro . ^-^-^ 

^Kiort, Creation, Deletion, Modification methods 
associated with broadcast objects - creation, 

and Notifications - for Custom Data Descriptors, 
unique ID that signifies the data type (or class) of the object: 

^^^^•■ w ^^, aU8 as this ID is used to determine the 
This ID must be guaranteed to be unique, as 

♦ «,hpn Custom Data notifications are received. The 
type of data being sent when custom 

,„ L,-i H +hP attributes and data for the 
35 next step is thus to define a structure to hold the attnbut 

custom data object. 

.tmct CMyCuatoaDataBlock 

1 cMyCuetwnPataBlock (); 

40 tat »_«*yCu»to»»«ril«tei 
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L^i-ct. ,c„at 0 cue e o»o bj .= t <, can be called to create a new custom object, 
set the data type of the Data Descriptor object, and set the attribute structure 
on the object: 

5 **y • 

^ //To create a Custom Data Object! nv»-i»~ 
// Fir" alk the Project to create a new Custom Data Obje- 
RktOfejectldType myCufltomOtojectld 
JO pProj act - >Croat e«CvB ^ [ n 

//.Get an interface to it 

eskt*tr< CRktCustaaObjeet > pCustonObject » 
^^□cSt8ervice B -»Cr..t.RktCu B to«0bjectIaterC. e a 

*)5 ( nyCustomObjectld )t 

// Create my custom data block and fill it in... 
CWyCuetoxoDataBlock myCuetomData j 



20 



25 



// Set the custom data type __. . . 

pCuBtomObjact->SeU)atBType( Mr^COSTOM^OBJECT^ID ), 

// Attach the extended data to the and * iZS) 

i nt 32Type nSixe - aiaeox < ™>^ 8 ^ t ^?^ 
pCustoaQbjeet->SetData< fcmyCuBtomData, nSise )j 



catch ( CRlctException e ) 
30 { 

e .ReportRktError ( ) t 

• } Wheaclientapplicatip^ 

object, it simply checks the data type and handles it as necessary: 

35 // To access an existing Custom Data Object: 
try 

// Assume we start with the ID of the object... 

40 // Get an interface to it 

CRktPtr< CRfctCustomObject > 

PC r^^ t t8:rvic.B.>Creat.«ctCu.to B Ob J .ct I nter t .ca 
* myCustomObjectld ) ; 

— ■ Yvt^^^^^ "- 

// but we'll be eafe 
50 RJctCustoaDataldType idCuetomi 

. . . idCustcm o 

} ; if ( idCustom «- CLSID_MY_CUSTOM_DATA ) 

cc ' // create my custom data scruct 

OSyCustomDataBlock xayCus tonData; 

// Get the Data. Upon return, theSiie is set to the actual 
// size of the data. 
50 Int32Type nSi.e . si*eof ( myCuetamData )> 
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5 . } // if roy custom data _ 
.iatc^^CBktBxcoptiont o ) 

* o.BoportWctBrrorl); . 

18 ' .. AI-i »«^ii*w«n!-.!» I *^" ,h ." w T 

- -component 24 (during se^ices opponent 24 inHialtzaton). Services 
. exponent 24 «, only allow creaOon en. reception o, custom <N* 
na ve been registered. Once registered, the date willbe downloaded 
15 automatically. 

_,_„■ .. services component 24 to send me ^"^^n obJECT_ID) » 

dps ofdata ma. reference recordadmadia. This recorded media is 
.presented by an Asset obiec, in tbe broadcast bbjec, mode) of data 

paging component 32. ******** '^present 3 

t ecorded compositional e^ment. U Is Vbese Asset obiects that are reterenced 

by clips to form arrangements. 

Though each Asset object represents a s,ng.e element there can be 
several versions o, the artua, recorded media for the object Thisallows 

• c^rthp Asset internal to the Asset each of 
users to create various versions of the Asset, im 

these versions is represented by a Rendering object 

Asset data is often very large and it is highly desirable for users to 
broad cas, compressed versions of Asset data. Because this compressed 
«. will often be degraded versions of me original recording, an Asset cannot 
simply repiace the original media data with the compressed data. 
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Asset objects provide a mechanism for tracking each version of the 
data and associating them with the original source data, as well as specifying 
which version(s) to broadcast to server 12. This is accomplished via 
Rendering objects. 

5 Each Asset object has a list of one or more Rendering objects, as 

shown in Fig. 6. For each Asset object, there is a Source Rendering . object, 
that represents the original, bit-accurate data. Alternate Rendering objects 
are derived from this original source data. 

The data for each rendering object is only broadcast to server 12 when 

10 specified by client application component 20. Likewise, rendering object data 
is only downloaded from server 12 when requested by client application 
component 20. 

Each rendering object thus acts as a placeholder for all potential 
1 versions of an Asset object that the user can get, describing all attributes of- 
1 5 the rendered data. Applications select which Rendering objects on server 12 
to download the data for, based on the ratio of quality to data size. 

Rendering Objects act as File Locator Objects in the broadcast object 
model. In a sense, Assets are abstract elements; it is Rendering Objects that 
actually hold the data. 
20 Renderings have two methods for storing data: 

In RAM as a data block. 
On disk as a File. 
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' The use o, RAM or d*x is large* based on .he s*e andlype of the 

nint Hatp is RAM-based, and audio 
data being stored. Typically, for instance, MID! data 

data is file-based. 

« a„ ob*c* in the broadcast 0*0 mode!, only Rendering 
5 .ached by cache module 36. Beca.se Rendering objec* are sen, fron, : 
se „er 12 on a regues.-only basis, services component 24 can checker 

Hf incal seauencer station 14 before 
the Rendering object is stored on disk of local sequ 

sending the data request. 

ln „. preferred embodiment Asset Renders object are limited «o 

10 three specific types: 

Source: Spec^.the origlna, soo-ce recording. Uerally -presents 

a-wt-accurate recreation of .he originally recorded file. 

Standard: Specifies the standard rendering o, me file to use, genara^y 
a moderate compressed version of the onginal source dafa. 
„ Preview: Specifies ,ne rendering «ra. shouid be downloaded in order 

toge ,apreviewo,themedia, 9 enera,,yahigh,ycompressedvers i0 no.m e: 

original source data. - • 

Each of the high-leve, Assa, calls se, forth in the Appendix uses a flag 
specifying which o, .he fhree Rendering obiec. types is being referenced by 
20 " ^ca, Typical* .he ,ype o, Rehdehng obiec. selected will be based on the 
type o,datacon l ainedby,eAsset. Simple date types - such as MIDI - will 
„„, use compression or a„erna.ive renderings. More complex data .ypes - 
such as Audio or Video -use a number of different rendering electa to 
facilitate efficient use of bandwidth- 
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A first example of use of asset objects will be described using MIDI 
data. Because the amount of data is relatively small, only the source 
rendering object is broadcast, with no compression and no alternative 
rendering types. 

5 The sender creates a new Asset object, sets its data, and broadcasts it 

to server 12. 

Step 1 : Create an Asset Object 

The first step for client application component 20 is to create an Asset 
object. This is done in the normal manner. 

10 // Attempt to Create an Asset in the current Project 
1U ^ectldtype ....tXd . pProJ.et-^reate^setO , 

Step 2: Set the Asset Data and Data Kind 

The next step is to set the data and data kind for the object. In this 

case, because the amount of data that we are sending is small, only the 

15 source data is set: 

^E-!22L2K--i. ( pMXWDace. nMXMD.taSi,. 

20 ' ' The setsourcenediao call is used to set the data on the Source rendering. 

The data kind of the data is set to to signify that the data is 

in standard MIDI file format. 

Step 3: Set the Asset Flags 
25 The Ihlrd step is to set the flags for the Asset. These flags specify ^ 
"which rendering of the asset to upload to the server 12 the next time a call to 

Br9 adca Bt( , is made. In this case, only the source data is required. 

// Always Broadcast MIDI 
Source 

30 j>MidiAitBet->SetBroadc»stFlag0 l 
ASEET_BROJVDCAST_SOiraCE ) t 
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Settng the .SET.BRO^T.s^ag specif.es that the source 
rendering must be uploaded for the object. 
Step 4: Broadcast 

The I as, step is «o broadcast. This is doners norma., in response to a 
5 command generated by the user : 

'•^Toraceive an Asset, client application components of local sequence 
sfcdon 14 handies the new Asset notation and requests me ease, data. - 
When me ^X^«**>»* received. «.e Asse, o bi eC hes 
heen created by data packaging module 26. Clien, application component 20 
ff ea,es an interface ,o me Asset 
available renderings : 

{ *~ ^iz:' -zzzz 

20 . / set an interface to the new asset 

c CWctM-et » PUMI * joctId) , 
Cr««teR3etXBoetInter£ace I ™J _ 

25 // wn8t "plaaeUoetDataXiudO I 

^° DataKlnciType dataJU.no. - P*s»» 

1.1... • ... ■:. I.;. -J ■' . - 

30 I ' „ „„, «» or w »"" "■" =q "" 

• v. . ... //. etc... ' I . r ,:.::v-- . n ( 

Le if « dataZiua - a«_«» ' 1 

35 ( „ CreBCe ene of my application- . Audio asset e^v 

// etc- 

) 

40 catch ( CWctBxception fce ) 

* a.BeportRktErrorO » 

} 



15 
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Data must always be requested by local sequencer station 12 for 
assets. This allows for flexibility when receiving large amounts of data. To do 
this client application component 20 simply initiates the download: 

virtual void 
5 CHyRltt9orvieoa:iOnAB80tMediaAvailablo ( 

const RktObjoetldTypefir rAesetld, 
const RendClaseType classification, 
const RKtObjectldfype* rRenderingld 

10 A ¥ 

try 

^ CRJctPtr < CRJetXssot > pAssat « 

CreateRktAaeetlnterface ( rAesetld )i 

15 

// Check if the media already exists on this machine. 
// If not; download it. (Note: this isn't necessarily 
// recommended - you should download media whenever 
// It impropriate. Your UI might even allow downloading 
20 // of assets on an individual basis). 

// Source is always Decompressed. 
// Other renderings download compressed. 
RendS tateType rendStatei ■ ■ 

25 if ( classification - ASSBT.SOOTCE J^^^B > 

rendS tate - AS BETDECOMPRBS SEI)_RBND_STATE / 

else mtB 

rondState - ASSJTr_COKPIUMSED_REHD_STATB 1 



Of) //if the media is not already local, then download it 

JU ( I pAssat- >XsHediaI*ocal ( classification. rendState ) ) 

{ // Note: If this media is RAM-based, the file locator 

// is. ignored. 
35 CRXtFilaLocator f iToLocDnuoed; 

pAssat- >DowaloadMadia 

( classification, f ileLocUaueod )t 

} 

40 catch ( CRktException fre ) 
^ e . RoportRxtRrror ( ) > 

When the data has been successfully downloaded, the 
45 onAssetMediaDownioadado Notification will be sent. At this point the data is 

available locally, and client application component 20 calls oetDatao to get a 
copy of the data: 

// This notification called when data has been downloaded 

virtual void , . . 

50 CMyRktServices: -.OnAssetMediaDownloaded I 

const RkcObjectldTypefi rAs'set.Id, 

const RendClassType £7d 22£ 

const RxtObjectldTypet rRenderingld cowr* 

55 try 

// Find my corresponding object 
CRktPtr « CRktAsset > pAsset - 

CreateRktAssetlnterface ( rAssetld ) ; 
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, 4 allocate a RAM based 
// Have services =°^" en L .„ che ,dacs in pPata 
// copy, and store a pointer -o cne 
// store its sise in nSiee. 

/, „„ t e: this application wiU be responsive for 
// freeing the. memory 



. void?. ..,pData.;. . 

10 iom 115126 : 



45 &PPata 
nSiie ); 

" - 1 " 



t>ABBet->GetMediaCopy ( 
ASSET SOURCE^REND^CLASS , 
ASSCT^DECOMPiESSED-REND-STATE. 



catch < CRHtException &e ) 
20 e. ReportRlct Error {); 



25 



!„ a second cample, an audio data Assets created. Can. 

■ ™ „K the audio data and a compressed preview 
application component 20 sets the auoio u 

rendering is generated automatically by services component 24, 

- -, ,„«s scenario the nar size is fl uKe large, so me data is eto.d .n a 



file 



The sender follows many of .he steps in .he simple MID, case above: 

used: 

on // »* the project »"•«•"" r * ^-"relteAsset ( ) 

30 ^^ eC "^^ 6 "the^ asset 



35 



^roer^SretrtheWasset 
( assetld ) ; 



^FUeLocator f ileLoc.tor , 
' ,/ set the f locator here (bring up a dialog or use,. - 
45 p^hnLe. Or use an FSSpec on) . 

pAseet-SetSourceMedie. * xileLocator-) ; 

pAsset- .setBroadcastFlage ( 



SUBSTITUTE SHEET (RULE 26) 



a* ,„,,«o PCMJSOO/25977 
WO 01/22398 

-40- 

AS SET_B ROAD CAS7_ PREVIEW ) ; 

// Make sure all renderings are created 
pMyRocketServiceB- >RenderForBroadcast < ) ; 

5 

// and Broadcast 
pMyRocketServices- >Broadcast ( ) ; 

Because ASSEr BRO^!^ST_PREyiEW yvas specified, services 
component 24 will automatically generate the preview rendering from the 
10 specified source rendering and flag it for upload when 

CRocketServices : : RenderForBroadcast ( ) Ss-caHed. 
Alternatively, the preview could be generated by calling 

CRktAsset: :COJnpreBBMedia() explicitly - . 

// compress the asset (true means synchronous) 
•15 pAB8et->CompressMedia ( 

ASSET_PREVIEW_REND_CLASS . T 

true ) ; 

In this example asset_broadcast_sotjrce was not set. This means 
20 that the Source Rendering has not been tagged for upload and will not be 
uploaded to server 12. 

The-source rendering could beaddedio ujp\^$mW&P^B&. ; 

r^S E ;1 R 0^rioUR?| 8 I ASSET_BR0ADCAS7_PREVIEW , , 
25 pMyRocketServices->Broa(Jcast() ; 

When an Asset is created and broadcast by a remote sequencer 
station 16, notification queue handler 28 generates an 
OnCreateAssetcoxnplete .( ). notification. Client application component 
30 than queries for the Asset ob ject, generally via a lookup by ID within its own 
data model: 

// find matching asset in my data mpdel. 
CMyAsaet-* pMyAsset = FindMyAsset ( idAsset ) ; 
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As above, the data would be requested: 

CRktFileLocator locDownloadDir ; 

// on Windows- - \\MyDownloadB\\" ); 

O /./ (similarly on Mac, but wouj. * 

FSSpec) pAsBet ->DownloadMedia( AS^.WBVIW . 

ftib^ownloftdpir;:):; _ .„,..•- . 

10 i^dMediaO specifies thexlassificationuof 

, j ,nH thP directory to which the downloaded file 
the rendering-data to download and the.d.rectory . . .. 

should be written. 

When .he data has been successfully downloaded, the 
16 ^^a^^nomcaeon- he sent. MMspoWUhe 

- w - -r — . — const .WctObjectwyp_ . £ iCBtlon - 

20 ....... V SKSS3BU— «• 



25 



try 

// Get an interface to the aaset 



V*. 

30 



35 



t / / wcw — 

r cRXtPtr < CRxtABset > P^ 88t . T; 
Create^tAssetxnterface t rAssetld ) 

7/ and get set the data Jg^J^^^.- m ). 
pAsset- >DecotnpressRenaering i 

v ! catch I CR)tt Except ion &e ) 

' e.ReportRfctErrorl) ? 

When the data has been successfully decompressed, the 
OnAssetDataDeco^reBsedO notffication will be sent: = 

„ Thl. -.ific.ll" — 

const RktObjectldTypehrn cla86 if ication, 



45 
50 



cry 

{ createRktAssetlnterface t rAssetld >, 

/, Get the Audio data for this asset to a file. 
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CRktPileU>cator locDecompresfledTile « 
pMyAsae t - >Get Media 
(classification., 
ASSET DECOMPRESSED REND_STATE ) t 
5 ?/ S= 9 iSort the file Specified by locDecompreesedPile 

// -into Application- 
catch t CRktBxception fce ) 

i H e . ReportRktError < > ; 

) 

*/ 

Services component 24 keeps track of what files ithas written to disk 
client application component 20 can then check these files to determine what 
1 5 files need to be downloaded during a data request Files that are already 
available need not be downloaded. Calls to isMediaLocal ( ) indicate if 
media has been downloaded already. 

Services component24 uses Data Locator files to track and cache data 
for Rendering objects. Each data locator file is identified by the ID of the 
20 rendering it corresponds to, the time of the last modification of the rendering, 

-and a prefix indicating whemer the cached data i preprocessed 
(compressed) or post-processed (decompressed ). 

For file-based rendering objects, files are written in locations specified 
by the client application. This allows media files to be grouped in directories 
25 by project. It also means that client application component 20 can use 
whatever file organization scheme it chooses. 

Each project object has a corresponding folder in the cache directory. 
" TikeData Locators, the directories are namedwiththe-iD of the project they- 
correspond to. Data Locator objects are stored within the folder of the project 
30 that contains them. 



SUBSTITUTE SHEET (RULE 26) 



FCIYUSOO/25977 • • 

-43- 

Because media ffles can take up quite a io. of disKspace.it is 

^ M a.* with the preview version ofan Asset, men laterxhoose «o 
, ^ n! oao .he source rendering. At this point the-preview-fendering is - 

// Clear up --he media «e are no longer using- 

" %^--assw-- ■ • 

ASSET COMPRESSSD.REHD STATE >: 

10..- ptaaet-^ele'.eLocalMedia 

I ASSET PREVIEW REKD_CLASS, . 

ASSET.DECOKPP.ESSBD RE«r jjM 1. from 

This call both clears the rendenng file from the cache a 

15 disk or RAM. 

be apparent to .hose sWed in me ar«ha. various moddlcahona 

v - and varies can he made in the methods and systems consistent with the 
Pre sen,invenuonw,thou,depa,n gfr om.hespi,.o r soopeo f u,e,nven tt on. 

F „ r exampie. if i o, the constants in aha invenson deschhed.eoove were 

• . ^ invention and woutd ha tuncuon* eguWalent. The true scope 
claims is defined by the following claims,, 
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1. 



Apparatus for sharing sequence data between a local sequencer 
station and at least one remote sequencer station over a network via a 
server, the sequence data representing audiovisual occurrences each 
5 having descriptive characteristics and time characteristics, the 

apparatus, compxlslrig; 

a first interface module receiving commands from a local sequencer 

station; 

a data packaging module coupled to the first interface module, the data 
1 o packaging module responding to the received commands by 

encapsulating sequence data from the local sequencer station 
into broadcast data units retaining the descriptive characteristics 
and time relationships of the sequence data, the data packaging 

module also extracting sequence data from broadcast data units 

1 5 received from the server for access by the local sequencer 

terminal; 

a broadcast handler coupled to the first interface module and the data 
packaging module, the broadcast handler processing 
commands received via the first interface module; 
20 a server communications module responding to commands processed 

" ~ by the broadcast hlthllerbytrari^itt^ to 
the server for distribution to at least one remote sequencer 
station, the server communications module also receiving data 
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available messages and broadcast data units from the server; 
and 

^notification queue handler coupled to fhe server communications 

module and responsive to receipt of data available messages 
ana broad cast data unitsfrom the serve* transmit notifications 

' ! to^nretinterta^foraccessby^localsequencerterminal. 

Apparatus as recited in claim 1 wherein the dare packaging module 
encapsulates the sequence data into broadcast data units including an 
arrangement dare unit establishing e time reference, and a. ieas, one 
track data unit having a track time reference corresponding to the : 
arrangement time reference, each track data unit having at least one 
associated even, data unit representing an audiovisual occarrence-at* 

. it tn the associated track time reference, 
specified time with respect to the assoucu 

Apparatus as recited in Calm 2, wherein the sequence date produced- 
by a. local sequencer station includes multimedia data source dare 
units and wherein the data packaging module encapsulates the 
nrukimedia source dare units into a, leas, one type of asset rendering 
broadcast unit, each asset rendering broadcast unit type specifying a 
version of multimedia dare source dare exhibiting a different degree of 
data compression. 

Apparatus as recited in claim 3, wherein the server communications 
m cdule responds to commands processed by the broadcast handler by 
transmitting asset rendering broadcast unite o, a selected asset 
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rendering broadcast unit type to the server for distribution to at least 
one remote sequencer station 

5. Apparatus as recited in claim 3, wherein the sequence data units 
produced by the local sequencer station include clip data units each 
representing a specified portion of a multimedia data source data unit 
andwherein the data packaging module encapsulates the clip. data 
units into broadcast clip data units. 

6. Apparatus as recited in claim 5, wherein the data packaging module 
encapsulates sequence data units into broadcast clip event data units 
each representing a specified portion of a multimedia data source data 
unit beginning at a specified time with respect to an associated track 
time reference. 

7. Apparatus as recited in claim 6, wherein: 

the data packaging module encapsulates sequence data units into 
scope event data units each having a scope event time 
reference established at a specific time with respect to an 
associated track time reference; 

each scope event data unit including at least one timeline event data 
unit, each timeline event data unit having a timeline event time 
reference established at a specific time with respect to the 
associated scope event time reference and including at least 
one event data unit representing an audiovisual occurrence at a 
specified time with respect to the associated timeline event time 
reference. 
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8 ; Apparatus as redted in claim 1, comprising a connection control 
component responsive to commands received from the local 
sequencer station to establish access via the server to a predetermined 
subset of broadcast oata units stored on the server. 
5 ' Apparatus as recned in daim8>hereir the conr«dion control 

' ' "and establishes access to a predetermined subse. of broadcast data 
. • uriits atored on the server in accordance with permission data stored 



10 



15 



20 



on the server. 

10 ; Apparatus es recited in claim 1, wherein the date-packaging modu-e: 
encapsulates sequence data Into first andsecondlypes of 

broadcast data units; 
responds to receipt of 

the server of the first type of broadcast data unit by 
causing the server communications module to initiate a 
download of the first type of broadcast data unit without 
requiring autho"rization from the client application 
component; and 
responds to receipt of a message indicating the availability at 
the server of the second type of broadcast data unit by 
' ^ causing the server communications module to initiate a 
download of the second type of broadcast data unit only 
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after receipt of a download command from the client 
application component. 
Apparatus as recited in claim 10, wherein the first type of broadcast 
data unit comprises a non-media broadcast data unit and the second 
type of broadcast data unit comprises a media broadcast data unit 

12. Apparatus for sharing sequence data between a local sequencer 

station and at least one remote sequencer station over a network via a 
server, the sequence data representing audiovisual occurrences each 
10 having descriptive characteristics and time characteristics and 

including multimedia data source data units, the apparatus comprising: 
a first interface module receiving commands from a local sequencer 
station; 

a data packaging module coupled to .the..first interface module, the data. 

1 5 packaging module responding to the received commands by 

encapsulating sequence data from the local sequencer station 
into broadcast data units retaining the descriptive characteristics 
and time relationships of the sequence data, the data packaging 
module encapsulating the multimedia source data units into at 

20 least one type of asset rendering broadcast unit, each rendering 

broadcast unit type specifying a version of multimedia data 
source data exhibiting a different degree of data compression, 
the data packaging module also extracting sequence data from 
broadcast data units received from the server; 
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- a broadcast handler coupled to-the first interface module and the data 
packaging module, the broadcast handler processing 

commands received via the first interface module; and 

' r : : ' T ? a server communications moduleVesponding to commands processed 
5 r \ by the broadcast-handler by transmitting broadcast data units to 

; CU ' toe server for distrlbutioh to at least pne;rempte sequencer 

• r £: 5^00, the server communications module also receiving 
^ broadcast data units via the server from the at least one remote 

sequencer station. 

10 13. Apparatus fcr sharing sequericedaWbetweenelocal sequencer ^ 

^ having descriptive characteristics and time characteristics, the 

"1 . "1 1 apparatus comprising: 

15 " * afirs. interface module receiving commands from a local sequencer 

station; 

• " V data packaging module coupled to the first interface module, the data 
packaging module responding to the received commands by 
encapsulating sequence data from the local sequencer station 
into broadcast dafif units retainihg*e descriptive characteristics 
and time relationships of the sequence data, the broadcast data 

units including custom broadcast data units, standard broadcast 
' data units expressing the hierarchy of sequence deta, and 
specialized broadcast data units including all attributes of 
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standard broadcast data units plus additional attributes, the data 
packaging module also extracting sequence data from 
broadcast data units received from the server; 
a broadcast handler coupled to the first interface module and the data 
5 packaging module, the broadcast handler processing 

commands received via the first interface module; and 
a server communications module responding to commands processed 
by the broadcast handler by transmitting broadcast data units to 
the server for distribution to at least one remote sequencer 
10 station, the server communications module also receiving 

broadcast data units via the server from the at least one remote 
sequencer station and passing the received broadcast data units 
to the data packaging module. 
14. A method for sharing sequence data between a local sequencer station 
1 5 and at least one remote sequencer station over a network via a server, 

the sequence data representing audiovisual occurrences each having 
descriptive characteristics and time characteristics, the method 
comprising: 

receiving commands via a client application component from a user at 
"20— : - a local sequencer station; 

responding to the received commands by encapsulating sequence data 
from the local sequencer station into broadcast data units 
retaining the descriptive characteristics and time relationships of 
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me sequence date.andtansmit^broaacastpBa units to the 
server for distribution to oUeast one remote sequencer station; 
f receiving data avSilible messages fem the server 
: • -responding to race*,, of data available-messages from the.server to 
5 transmit notifications to me dient application component; 

: " responding jcoommanris received from the client application J. 
^ " ^ component to request download of broadcast data units from the 
server; and 

. reiving broadcast data units from the server and etfratfng sequence 
- - date from the received broadcast date unite for access b y the . 
dient application component. - 
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